Yo! Briefcase: index

Bashing Ambiguity

Thu Oct 19 00:00:00 UTC 2017

I read a great tweet the other year,

Was hoping to get some verbose output from pkill, ran pkill -9 -v process_name. Ho boy. — Maxime Chevalier (@Love2Code) September 21, 2016

I can't say I've ever used pkill so had to do a tiny bit of digging through the man pages of pkill

PKILL(1)                  BSD General Commands Manual                 PKILL(1)

NAME
     pgrep, pkill -- find or signal processes by name

SYNOPSIS
     pkill [-signal] [-ILafilnovx] [-F pidfile] [-G gid] [-P ppid] [-U uid] [-g pgrp] [-t tty] [-u euid] pattern ...

DESCRIPTION

     The pkill command searches the process table on the running system and signals all processes that match the criteria given on the command line.

     The following options are available:

  	 ...

     -v          Reverse the sense of the matching; display processes that do not match the given criteria.

So heh, pkill is based on pgrep which is based on grep and -v is the shorthand notation of --invert-match so the previous command becomes kill everything (with extreme prejudice) that isn't named process_name. Ho boy indeed. Of course it's an easy mistake to make, I'm certain I would have fallen prey of this eventually had I not seen this tweet. I'm known for simply throwing more than a few -vvvvvvvvvvvvv's onto commands that don't behave as I'd expect.

But AHA! I have the solution - just be more explicit and use the -- notation

Well.... not quite.

For some crazy bizarro reason pkill and pgrep don't have the double dash equivalent of v/invert-match. I'm sure there is a sane and logical rationale for this but then again, given it's computers all the way down, maybe not.

It's shit isn't it? Even the most standard tools we use as developers take the principle of least astonishment and just throw it into the endlessly burning tyre fire of misery and terrible user experience.

So what can we do? Easy! TRUST. NOTHING. Never assume anything. Slow down. Think. Think even when it seems obvious. Think even when you've written the same command for the 100th time. If you can't think or don't want to think make sure everything you do isn't irrevocably destructive, make sure you can get back to the same point you where at before you went and destroyed the world in a raging hellfire of accidental typage.

How to do unnecessary things in long winded ways...

Wed Jun 21 00:00:00 UTC 2017

Quick and Dirty JSON Builder for Java

Fri Feb 10 00:00:00 UTC 2017

If you're writing tests involving JSON or trying to return some basic JSON structure in a simple Play or Spark application here is a simple shorthand notation that can be used to build said response.

Given the following JSON structure,

{
  "title" : "Test Schema",
  "type" : "object",
  "properties" : {
    "firstName" : {
      "type" : "string"
    },
    "lastName" : {
      "type" : "string"
    },
    "age" : {
      "description" : "Age in years",
      "type" : "integer",
      "minimum" : 0
    }
  },
  "required" : [ "firstName", "lastName" ]
}

It is possible to represent this in Java like this,

toJson(of(
    "title", "Test Schema",
    "type", "object",
    "properties", of(
        "firstName", of("type", "string"),
        "lastName", of("type", "string"),
        "age", of(
            "description", "Age in years",
            "type", "integer",
            "minimum", 0
        )
    ),
    "required", new String[] { "firstName", "lastName" }
))

In order to achieve this then 2 static imports must be used,

import static com.google.common.collect.ImmutableMap.of;
import static play.libs.Json.toJson;

Obviously, this assume the use of Guava and Play but we can at least avoid the use of Play (i.e. if you're using some other web framework) by shimming the toJson method using the Jackson ObjectMapper directly e.g.

public JsonNode toJson(Object data) {
    // either provide or create an ObjectMapper
    return new ObjectMapper().valueToTree(data);
}

It's nicer than using, and having to wrangle, a massive string blob because it helps avoid some of accidental syntactical JSON errors that you can make (extra commas, unbalanced braces etc.) when dealing with a raw string. It's also a bit terser than using the Jackson builder directly as that can make deep chaining a bit awkward from what I've experienced.

CAVEAT: I use this in tests and basic webapps were performance is not important in the slightest so I haven't attempted to understand the overhead of using this approach.

ansible-console: An Interactive REPL for Ansible

Tue Jan 10 00:00:00 UTC 2017

If you know about ansible-console you can ignore this. If you don't this might be of use to you.

Something found out recently is that Ansible has an interactive REPL of sorts in ansible-console for doing some adhoc things on a collection of hosts.

Using it is pretty simple

> ansible-console -i preproduction

Welcome to the ansible console.
Type help or ? to list commands.

jamhughes@all (3)[f:5]$

This will bring you into the shell itself,

the -i preproduction tells the console to use a certain hosts file (in my case preproduction).

From here you can use the normal ansible modules. So for example if I wanted to get the date on all the hosts I can do this,

jamhughes@all (3)[f:5]$ shell date
frontend1.preproduction | SUCCESS | rc=0 >>
Tue Jan 10 22:04:18 GMT 2017

frontend2.preproduction | SUCCESS | rc=0 >>
Tue Jan 10 22:04:18 GMT 2017

backend1.preproduction | SUCCESS | rc=0 >>
Tue Jan 10 22:04:18 GMT 2017

If you have specific groups you want to work in you can cd into groups

jamhughes@all (3)[f:5]$ cd frontend
jamhughes@customer (3)[f:5]$ shell date
frontend1.preproduction | SUCCESS | rc=0 >>
Tue Jan 10 22:04:18 GMT 2017

frontend2.preproduction | SUCCESS | rc=0 >>
Tue Jan 10 22:04:18 GMT 2017

I guess it flies in the face of all this immutable infrastructure stuff but hey we are not all there yet and stuff like this can be handy.

Tabularizing output of shell scripts with column

Fri Jan 06 00:00:00 UTC 2017

I tend to live in my terminal and as such tend to write a bunch of shell scripts for various project specific things. One script I have lists, form a project directory, the what branch each of my repositories is on. It looks a bit like this,

#!/bin/bash

projects=(
  super-service 
  super-client 
  authentication-service 
  infrastructure    )

for project in ${projects[*]}; do
  cd $project
  echo "$project $(git branch | sed -n -e 's/^\* \(.*\)/\1/p')"
  cd ..
done

When I run this the output looks like this,

super-service master
super-client master
authentication-service experimental
infrastructure master

The list is longer and the names above aren't real but you see the problem. It's hard to see at a glance what repos are on what branches.

Enter `column`

If you pipe the output of the script above in column -t -s' ' the result is a bit different.

super-service           master
super-client            master
authentication-service  experimental
infrastructure          master

You can see from the output above the column command pretty much does what is says on the tin. It turns output into formatted table with discernible columns. This makes it much easier to read. The additional parameter I passed where,

-t which tells column to read the whole input and determine the number of columns automatically to print the table
-s' ' specifies the character to use to determine a column break when using the -t command. In this case, we specify whitespace. In fact whitespace is the default character to break on so this is actually not needed in our case but if each column was separated by a | character we could use -s'|' to change the default behavior.

Now go forth and prettify those scripts, they're worth it.

Functional Collections in Scala (Lightning Talk)

Tue Nov 08 00:00:00 UTC 2016

Kotlin - 2 Years On

Wed Feb 24 00:00:00 UTC 2016

I first played with Kotlin back in 2012, I've written some Android apps with and without it and generally played around with it. Since the release of Kotlin 1.0 a few days ago I've decided to jump back into it and see if my original views still hold up.

Herein lies opinion of the personal kind. Proceed with an open mind.

Scala is my day job, not only Scala but legacy/old Scala. Not only that but a lot of legacy/old Scala. Worse still not only a lot of legacy/old Scala but a lot of other peoples legacy/old Scala. Thanks to the feature rich and impressively flexible nature of Scala this means there is a lot of sharp edges, unpredictability and a fresh arduous learning curve for each functional area. It can seem like a language designed by Atë at times.

Java is no better of course, it just resides at the other end of a spectrum, with its anaemic yet rigid feature set and almost pointless type system it's no wonder you see so many over-engineered and baklava-class layered solutions. Yeah Java 8 has improved the language a fair bit but the sacrifices made for backwards compatibility has meant those changes don't often go far enough.

Both Scala and Java require a lot of work to simplify so there has to be a middle ground and I (still) think that middle ground is Kotlin.

Think of Kotlin as C# for the JVM, Scala the Good Parts, Java++ or simply a decent general purpose language that won't require the blood of your first born. It's created by Jetbrains, it's about 5ish years old and just gone version 1.0. It is used extensively on Jetbrains products, being hailed as "Swift for Android" and works seamlessly within a mixed code base. It offers features similar to C# Extension methods, Scala implicits, Scala case classes, multiple inheritance and solid collection functionality like LINQ. It doesn't just ape Scala and C# features but introduces relatively original concepts like delegated properties, class delegation and typecasting.

I'd also proffer that it has an easier transition from Java to Kotlin than it is for Scala though I have no evidence for this other than my own experience of all 3 languages (including leading mixed skill teams in Scala and Java).

You could ask "but why not one of the other many languages available on the JVM?" and that would be fair. Many JVM languages haven't seen a great deal of traction or remain relatively niche but I think Kotlin is much better placed. It's more "general purpose" than a lot of the alt languages which means it will map better to current practises. It's closer to Java and/or Scala than other languages, it strikes the right balance around the type system with features like type inference and enhanced generics and it doesn't hurt that Jetbrains; a major player in the JVM world; is heavily invested in Kotlins future.

In the two years since I started playing with Kotlin it has aged well. It resisted adding a plethora of features for the sake of it and instead created a core syntax and feature set that allows developers enough wiggle room for being creative without turning them into Wizards of Arcane DSLs. This is a sweet spot for me when it comes to building and supporting a service over many years as it avoids the unnecessary pain that comes from the extreme ends of spectrum that Java and Scala tend to reside. Of course this doesn't mean it's perfect, nothing is. You're still adding another language to a project and taking on the associated baggage that comes with it like tooling differences and additional upskilling. Its not suddenly going to make your organisation move the JVM if they aren't already there. It's also not going to fix all your problems - bad architecture will remain bad architecture and bad patterns will remain bad patterns. However, with that said, I believe Kotlin will reduce enough of the friction of general software development and maintenance on the JVM to make it worth the investment.

alias, union and case in Crystal

Thu Jan 14 00:00:00 UTC 2016

So I've been diving into Crystal lately. It's nice. This isn't a post about why I think it's nice, it's a post about a few of the features of Crystal so let’s get straight down to it.

`alias`

alias is a very common feature that we can see across many languages but it can be used to build upon other features in Crystal. As the name suggests you use alias to give a type a different name.

alias TableOfContents = Hash(String, Int32)

In the example above we give the type Hash(String, Int32) a custom alias of TableOfContents which allows us to refer to this type as TableOfContents

def lookup(chapter : String, tableOfContents : TableOfContents) : Int32
  tableOfContents.fetch(chapter)
end

So instead of using Hash(String, String) in the method signature above we can use TableOfContents. So why is this useful? Well for one thing it allows us to reduce complex type names into smaller names. The example above isn't exactly verbose but longer types can be more readable.

Now of course size isn't everything so another advantage of alias is that we can add meaning to types within our problem domain. You're less likely to wonder what this hash of string to string is or should contain when we give a name like TableOfContents. In fact, we could enrich the signature above more by giving specific types an alias.

alias ChapterTitle = String
alias PageNumber = Int32
alias TableOfContents = Hash(ChapterTitle, PageNumber)

From this we can infer that for a given chapter title the table of contents is capable of finding the page number that the chapter starts on.

`union`

Unlike many languages the return type of methods can actually be more than one type. Crystal uses type inference so declaring types is not mandatory. Take a look at this,

def doAThing(isAThing : Bool)
  if isAThing
    "I did a thing"
  else
    42
  end
end

What type is this? It could be an Int32 it could be a String. In other statically typed languages the type most likely resolves to some common base class like Object or Any which makes sense but is actually a tiny bit pants ¹.

In Crystal this type will be inferred as String|Int32. This is a union and is essentially a type of its own that could be either a String or a Int32. We can explicitly give this method a return type which means the compiler can verify our assumption

def doAThing(isAThing : Bool) : String|Bool

You may have noticed I just wrote the wrong return type and attempting to compile the code results in a helpful compilation error message

Error in ./src/something.cr:14: instantiating 'doAThing(Bool)'

doAThing(false)
^

in ./src/something.cr:6: type must be (String | Bool), not (String | Int32)

def doAThing(t : Bool) : String|Bool
    ^

Union types give us the rigidity and predictability of types while, at the same time, allowing a certain amount of flexibility as if we were working in a purely dynamic language.

`case`

So we've got these nice union types how do we deal with them in code? Well similar to other languages Crystal has methods like is_a?(type_name) and responds_to?(method_name) that allows us to inspect the current type of the variable but it also supports a kind of pattern matching for types using case.

def readFile(filename : String) : String|Error
  # read file return a string of its contents
rescue an_error
  an_error
end

maybeContents = readFile("somefile.txt")

case maybeContents
when String
  puts maybeContents
when Error
  puts "We had a wee error doing that"
end

In each when of the case the type of maybeContents will be the specific type that was matched against. So maybeContents will start as a String|Error but be available as a String in the first when and an Error in the second when.

^{1: "pants" means bad.}

entr

Wed Jan 13 00:00:00 UTC 2016

entr is a command line utility for running arbitrary commands when files change.

Just for clarification - I did not contribute to this project. I just happen to find it useful.

Yes, yes you already have all this with guard, live-reload, guardian, watcher, watchy, watch, bouncy, bounce, bouncer, sir-loads-alot, loady, loadup, springy, trampoline, hot-rocket ¹ etc. but entr is worth a look. Why? Well it has a few criteria that make it compelling,

It's not locked into a particular use case, language or ecosystem
It has no external dependencies
It works on more than just Linux
It's command line friendly (works with pipes)

So lets just see it in action,

Install it

You can either build it from source or if you're using a mac you an use brew

brew update && brew install entr

Use it

What you want to do is get a list of files you are interested in i.e. ones that should trigger "something" when they change. This can be something as simple as ls or as convoluted as find . -iname '*.cr' -not -path './.crystal/*' something like. One way or another a command that spits out a file list is the first part.

Next you pipe this into entr with the command you want to run.

ls -d * | entr make

In this case when a file listed by ls changes make will be ran.

Further betterness

This is good enough for many use cases but there are always edge cases. One good example is live reloading a web server. In this case the process that you spawn typically blocks until killed which would cause issues with the above command. To cater for this use case entr gives us the -r flag.

ls -d * | entr -r lein ring server-headless

With the -r entr will attempt to terminate then running process prior to starting it again.

We also have the -c option. This does nothing more than clear the screen before running the command.

Finally, entr provides the ability to pass a special argument, /_, into the command it is configured to run. This argument will be replaced with the path to the first file that has changed. This is useful when you specifically need to execute the changed file, for example, when working with data loading scripts,

ls ./data/*.js | entr mongo < /_

entr is nice and simple with no external dependencies and not specific to any particular technology stack.

^{1: I may have made some or all of these up.}

An example of refactoring Scala code

Sat Jan 02 00:00:00 UTC 2016

I recently had to add code to an old and critical area of a 2 year old, rather large, Scala project. The code which centred around a core function that accepted an object instance, performed various actions that ultimately updated/transformed some properties on the instance. The passed in instance, a case class, used immutable values for all its properties so in order to apply these additional transforms the function created additional internal values after each transformation. Finally, it would return the final updated instance.

My work involved introducing yet another step in this series of transformations so I decided to play with various ways of achieving the same thing to see if there was a better way of approaching the problem.

Set the scene

The following code is contrived for the purposes of this article. It may seem overly simple but I didn't want to bog down the examples with more than necessary.

So let's set the scene using a stripped down example.

First of all, we have a primary Order model. An order instance is created and persisted once the customer has completed the order and This has a number of optional properties that will be set as the order is processed through various states (new order, order completed, order paid etc.)

case class Order(
    id: String, 
    paymentMethod: PaymentMethod,
    requisitionRequest: Option[RequisitionRequest], 
    invoice: Option[Invoice],
    paymentId: Option[String], 
    fulfilled: Boolean, 
    processedTimestamp: Option[Long])

PaymentMethod is a trait that a number of case objects extend,

sealed trait PaymentMethod
object PaymentMethods {
  case object Cash extends PaymentMethod
  case object CreditCard extends PaymentMethod
}

RequisitionRequest and Invoice are empty case classes that act as stubs. Their content isn't really important for these examples.

In our example we have a core function that will process an order after payment has been completed.

def processOrderPostPayment(order: Order): Order = {
	???
}

This function should perform a number of things

It should submit the order to the stock room and update the order with a requisitionRequest
If the order is paid by cash it should generate a new invoice and add this to the order (customer has not yet actually paid but agreed to pay)
If the order is paid by anything other than cash then it should retrieve the payment ID from the payment system and update the order
Once these things have completed it should mark the order as fulfilled, and finally,
It should update the order with the processed date.

We can set up some stub functions,

def submitToStockroom(order: Order): Order = {
  order.copy(requisitionRequest = Some(RequisitionRequest()))
}

def generateInvoice(order: Order): Order = {
  order.copy(invoice = Some(Invoice()))
}

def retrievePaymentNotification(order: Order): Order = {
  order.copy(paymentId = Some("payment"))
}

def markOrderProcessed(order: Order): Order = {
  order.copy(processedTimestamp = Some(1451468923165L))
}

Let's assume that we are adding the additional functionality of updating the processedTimestamp as this was, for some reason, not being done before.

The current solution

The existing solution looked a bit like this,

def processOrderPostPayment(order: Order): Order = {
  val submittedOrder = submitToStockroom(order)

  val updatedOrder = submittedOrder match {
    case o@Order(_,PaymentMethods.Cash,_,_,_,_,_) =>
      generateInvoice(o)
    case o =>
      retrievePaymentNotification(o)
  }

  updatedOrder.copy(fulfilled = true)
}

After a first pass of adding the new functionality of adding the processed timestamp it looked like this,

def processOrderPostPayment(order: Order): Order = {
  val submittedOrder = submitToStockroom(order)

  val updatedOrder = submittedOrder match {
    case o@Order(_,PaymentMethods.Cash,_,_,_,_,_) => 
      generateInvoice(o)
    case o => 
      retrievePaymentNotification(o)
  }

  val fulfilledOrder = updatedOrder.copy(fulfilled = true)

  markOrderProcessed(fulfilledOrder)
}

My first pass was to follow a similar pattern of assigning an updated version of the order to a val and subsequently pass that to the new markOrderProcessed function.

I think it's worth pointing out again that, yes, there are ways to make even this style a bit terser but instead focus on the overall style rather than the specifics of this example

There are a few things that bugged me about this approach. While it was consistent and rather easy to update it did cause some issues.

The actual code was more complex and it took some understanding where to place the actual code. Subsequent changes would further increase this complexity.
Coming up with a meaningful name for the new value was frustrating and I simply refuse to just append a number to a values name.
I introduced a bug because at one point I referenced the wrong transient value.

Beside the bug that, thankfully, a unit test flagged up none of this is exactly show stopping but I was curious if we could improve this function.

One more caveat. These examples are not necessarily better. This is not an article about gradual refactoring to a better solution. Each example is an alternative way of expressing the same problem and each example has its flaws. Consistency of approach across code is a godsend when trying to dive into legacy code.

Why not just use a `var`?

Instead of using new val declarations for each step of the transformation why not use a var?

def processOrderPostPayment(order: Order): Order = {
  var updatedOrder = submitToStockroom(order)

  updatedOrder = updatedOrder match {
    case o@Order(_,PaymentMethods.Cash,_,_,_,_,_) =>
      generateInvoice(o)
    case _ =>
      retrievePaymentNotification(updatedOrder)
  }

  updatedOrder = updatedOrder.copy(fulfilled = true)

  markOrderProcessed(updatedOrder)
}

I've interacted, both directly and indirectly, with many people that think having even a single var in your code base is an affront to the Scala gods but more times than not this appears to be down to lack of rationale thinking. Yes, we've created mutable state but we are still dealing with immutable object and any mutation is confined to this single function - local mutable state can be acceptable.

What this approach does is reduce the number of variable/values used to two

The order that was passed in, and,
The updatedOrder that we create and assign to multiple times

This tactic reduces the chances of using the wrong value in a function call compared to creating multiple values in the previous approach.

Why do we even need the `var`?

So if reducing the amount of variables/values we create appears to have a positive effect - why not simply never create one in this function?

def processOrderPostPayment(order: Order): Order = {
  markOrderProcessed((if(order.paymentMethod == PaymentMethods.Cash) generateInvoice _ else retrievePaymentNotification _)(submitToStockroom(order)).copy(fulfilled = true))
}

Yikes! So lets break this down. Based on the paymentMethod we are creating a partial function based on either generateInvoice or retrievePaymentNotification. From here we are passing in the result of submitToStockroom(order), updating the object using the copy constructor to mark it fulfilled and finally marking the order as processed via markOrderProcessed.

If I had to deconstruct this as someone else's code I'd probably need to take the rest of the day off to recover. Please never do this.

Can we improve this approach though?

def processOrderPostPayment(order: Order): Order = {
  markOrderProcessed((submitToStockroom(order) match {
    case o@Order(_,PaymentMethods.Cash,_,_,_,_,_) =>
      generateInvoice(o)
    case o =>
      retrievePaymentNotification(o)
  }).copy(fulfilled = true))
}

Here we have dropped the currying of functions in favour of simply nesting invocations using a match. Is it any better? Maybe. I think it's a tiny bit easier to decipher but it is still a big ball of tangled code. The order of execution doesn't flow from top to bottom but instead from inside out, kind of. This makes it very hard to derive execution order and so updating this code, especially when order is important, would be a massive pain.

Improving the flow

I've written a fair bit of Clojure over the last 2 years and one of the feature I really like about it is the threading macro. It's a tiny bit of syntactic sugar that would let you unroll calls like this,

// BTW this is not clojure it's scala-ish pseudocode
head(split(replace(toUppercase("abcd"), "A", "X"), ""))

Into something like this,

// again this is scala-ish pseudocode
thread("abcd", toUppercase, replace(_,"A","X"), split(_,""), head)

This more clearly defines the order of execution to the reader.

Starting with the string "abcd"
Make the string uppercase
Replace "A" with "X"
Split by character
Get the first character

With the first approach you are forced to work from inside out to understand what is happening and in what order.

Scala doesn't have a standard threading operator or macro but it does have some ways to accomplish something similar.

Using `Option` as a pipeline

You can think of an Option as a list of zero or one items that you can apply operations on. So if we make a Some(Order) based on the order passed into the function we can apply operations on the contained order in a way that mimics the thread operator.

def processOrderPostPayment(order: Order): Order = {
  Some(order)
    .map(submitToStockroom)
    .collect {
      case o@Order(_,PaymentMethods.Cash,_,_,_,_,_) => generateInvoice(o)
      case o => retrievePaymentNotification(o)
    }
    .map(_.copy(fulfilled = true))
    .map(markOrderProcessed)
    .get
}

Here we are using the power of map and collect to transform the original order through a series of steps without having to create intermediate variables or values. Finally we call get to unwrap the Order again. This approach only really works if we are always at least accepting and returning an Order.

Compose a pipeline

Using Some is alright I guess but for this type of problem Scala does have something we can use to create a pipeline - functional composition.

def processOrderPostPayment(order: Order): Order =
  (submitToStockroom _)
    .andThen(if (order.paymentMethod == PaymentMethods.Cash) generateInvoice else retrievePaymentNotification)
    .andThen(_.copy(fulfilled = true))
    .andThen(markOrderProcessed)(order)

In this example we are currying or partially applying the submitToStockroom function which produces a new type scala.Function1[Order, Order] which is a function that accepts a single Order and returns an Order. This gives us access to the compositional functions of the Function1 type - specifically andThen in our case. Using andThen we compose a single function that pipes its input and output through the series of functions in order. Finally we execute the function by invoking it with our input order (order).

So which is best?

This was never about one approach being better than another. There are a few examples that are clearly not reasonable solutions and would in fact make everything worse but there is no clear winner.

As you can see there are many ways to skin a cat. Depending on the context one approach may yield code that is easier to understand and maintain.

Refactoring Pattern Matching in Scala

Mon Dec 21 00:00:00 UTC 2015

I want to take a look at making effective use of Scala's pattern matching capabilities for a fairly trivial example. Don't expect any of this to be mind blowing but maybe you're venturing in Scala for the first time or have been tinkering for a while you might find this useful.

The example I give below has been taken from a real world example but the context has been changed and made bit more trivial and non-specific.

Setting the stage

We have a solution by which a customer can place an order through our system. They can pay by various means - Credit Card, Bitcoin and Direct Debit. Once they have placed their order they are taken to a payment selection screen where they can select how they want to pay. Finally, once they have selected their payment method they are redirected to the appropriate payment gateway.

First lets model the PaymentMethod types

sealed trait PaymentMethod

object PaymentMethod {
  case object Card extends PaymentMethod
  case object Bitcoin extends PaymentMethod
  case object DirectDebit extends PaymentMethod
}

This gives us case objects that represent the payment types our system supports. Next lets create an Order class to model our actual order.

case class Order(id: String, total: BigDecimal, selectedPaymentMethod: Option[PaymentMethod])

This class has some properties relating to the order

An id property, and
A total property that holds the final price of the order.

It also has the selectedPaymentMethod property which is an option of our PaymentMethod type. It is defined as an Option because prior to payment selection we will have an order but no payment method selected.

Now lets assume our system is an event based system. Once payment select has been made the Order instance is persisted to a data store with the chosen option and a system event is triggered to being processing payment and routing the customer.

Within whatever event handling strategy we use (Actor, Queue, Bus, Callback etc.) we then need to retrieve the customers most recent order and if there is a payment method defined route to the correct payment gateway. We have defined a method like for retrieving the mot recent customer order,

def mostRecentCustomerOrder(customerId: String): Option[Order]

It returns a Option[Order] because there are situations where this is used and the customer doesn't yet have a recent order. Just go with it.

Decisions using pattern matching

Now to the juicy bit. Given we've got the most recent order we need to route accordingly. At this point we want to return yet another Option this time of a potential Route which is an abstraction that allows us to route to external systems (the details are really not important for this example).

If there is no order we return None
If there is an order but no selected payment method we return None
If there is an order and a selected method we return a Some of a route depending on what the select method is.

Lets code this up.

mostRecentCustomerOrder(customerId) match {
  case Some(order) if order.selectedPaymentMethod == Some(PaymentMethod.Card) =>
    Some(redirectToPaymentGateway())
  case Some(order) if order.selectedPaymentMethod == Some(PaymentMethod.Bitcoin) =>
    Some(redirectToBitcoinGateway())
  case Some(order) if order.selectedPaymentMethod == Some(PaymentMethod.DirectDebit) =>
    Some(redirectToBankGateway())
  case _ => None
}

I see this code a lot. It's not necessarily wrong but it is rather long winded. We are doing some things unnecessarily. For example, we are extracting the Order into an order value and inspecting the contents. Not a major issue but we aren't actually using it inside the case statements body. We also run the risk of shadowing other values of the same name.

Deeper matching

Pattern matching in Scala isn't confined to the top level class. It is entirely possible to match all the way down the object graph. So lets revisit the same bit of code and remove the unnecessary extraction of a value and just use pure matching.

mostRecentCustomerOrder(customerId) match {
  case Some(Order(_, _, Some(PaymentMethod.Card))) =>
    Some(redirectToPaymentGateway())
  case Some(Order(_, _, Some(PaymentMethod.Bitcoin))) =>
    Some(redirectToBitcoinGateway())
  case Some(Order(_, _, Some(PaymentMethod.DirectDebit))) =>
    Some(redirectToBankGateway())
  case _ => None
}

In this case we avoid extracting the order value and make each case a bit more succinct by matching purely on the existence of a PaymentMethod within the Option. We also use _ to mark values we aren't interested in. Whether or not this is easier to read than the previous example is very subjective. I think it is but only really marginally so.

Option.collect

There is another option (pun intended). As mostRecentCustomerOrder returns an Option and we are expected to return an Option we can use Option.collect.

collect[B](pf: PartialFunction[A, B]): Option[B]

Returns a scala.Some containing the result of applying pf 
to this scala.Option's contained value, if this option is 
nonempty and pf is defined for that value.

This gives us the power of pattern matching so we can transform the input Option in to what we want if and only if the input satisfies our conditions, falling back to None if it doesn't. You can think of collect like a conditional map

mostRecentCustomerOrder(customerId).collect {
  case Order(_, _, Some(PaymentMethod.Card)) =>
    redirectToPaymentGateway()
  case Order(_, _, Some(PaymentMethod.Bitcoin)) =>
    redirectToBitcoinGateway()
  case Order(_, _, Some(PaymentMethod.DirectDebit)) =>
  	redirectToBankGateway()
}

By using collect we don't need to provide the default fallback case of returning None plus we get to unwrap all those Some(Order(...)) cases as well as not having to wrap each cases return value in a Some.

Again this is purely personal opinion but this reduction of noise makes this code much easier to read. We are letting the language deal with the default cases without having to repeat ourselves.

Summary

None (pun intended) of the options (pun intended) above are necessarily wrong and one approach may fit better in some scenarios than another but I feel that laying out the options (pun intended) helps us make more informed decisions while we try and build quality code.

On Popularity

Fri Nov 27 00:00:00 UTC 2015

Overhearing two developers lamenting someone’s choice of library it seemed that after bit of googling they deemed the library wasn't very good. It didn't have enough stars on GitHub.

Popularity is an awful metric to judge anything on and the data behind thumbs or stars or hearts is so skewed to be meaningless too.

Birth of a Star

Having released a few inconsequential open source projects on GitHub personally I noticed that stars arrived in for my projects shortly after the project was announced or when it got linked to by places like Twitter or Reddit. There was no way people actually dropped everything they where doing, used my code and immediately found benefit. So this begs the question - what does a "star" mean?

"Hey thanks this looks great and I may use it soon" - LIKELY
"I must remember to look at this when I have more time" - LIKELY
"Hey look at me I may or may not have used this popular library, accept me" - POSSIBLY
"What does this button actually do?" - I GUESS SO
"After 6, long, months I've just shipped a project using this library and it helped me avoid hours of pain" - UNLIKELY

The actions of "starring" means different things to different people but given the fact that they shoot up rapidly after the repo gets featured on an influential site suggests that there are certainly a lot of people using it as an acknowledgement that the project looks interesting. Now that’s not to say that early star adopters never use the project - many popular projects clearly have a large active user base but it's not because of the number of stars they have.

Death of a Star

So you visit a new project on GitHub, recommended to you by someone on TwitteRedditNews. You read the README.md and think - "cool". You hit the little star and download the project. You start tinkering with it, maybe write a blog post about it. Perhaps even use it in a small side project. Then you get distracted by life. You move on to other projects using different technologies and that, once cool, library is forgotten.

How many people actually actively go back and prune their "stars"? I'm willing to bet not many. Why bother, right? Staring is just a meaningless gesture yet people still insist on judging a project based on how many people performed these meaningless actions.

Heck I imagine even if you got badly burnt by a particular project after a few months of using it you'd probably still not remember to go back and unstar it. I presume it would take a public outcry and a good old internet pile-on to make a significant dent in a projects "star" popularity

Starry Starry Night

There are many ways to determine the viability of a project but "stars" isn't one of them. Sure you could use it to determine if one project is more popular than the other but what does that actually tell you? Without other (actually useful) data it means very little. Outside of comparing similar projects it means even less - some genuinely useful projects have little to no stars because they don't do "popular" things, just; you know, stuff that can really make a difference.

Stop look and listen. Oh and think as well. Do lots of that.

Popularity is not a indicator of quality (though quality may build popularity).
Popularity is not an indicator that you'll get meaningful support or documentation (though good support and documentation may build popularity)
Not all good things are necessarily popular/shiny (accessibility, security etc. - all those important things that just aren't all that fun but absolutely essential)
Indicators for popularity don't necessarily mean the subject is still popular
Relative popularity could be down to better marketing, shouting louder or simply luck.

Stars, hearts, thumbs up, pluses, smiles, artisanal breadsticks or whatever else we use to suggest the presence of the magical unicorn we call brand engagement - ignore them. Make informed decisions with real data specific to your needs.

Is it more work?

Yes.

All worthwhile things are. Thats why they are worthwhile.

The Twelve Factor App (Redux)

Sun Sep 20 00:00:00 UTC 2015

More Betterness

Wed Sep 09 00:00:00 UTC 2015

This article is adapted from a presentation I gave on team work and understanding

Hi. This wont take very long so just sit back and enjoy yourself. I want to start with a quote.

Have a read of that, and let it sink in for a moment before we move on. We'll come back to it later.

OK? Good.

I'd like to introduce you to a few people.

This is Alice. Typical hacker, Alice rarely writes tests. Never mind Test Driven Development there is literally no tests. Talk about pushing more broken garbage out in to the world. Boo Alice. Boo.

Here's Bob. Bob bought a book recently a learned the "Boy Scout Rule". Since then he has been helping his team out by making old code more concise. Bob, you're awesome.

Hi Caitlin. Caitlin's team are a bit peeved at Caitlin because even though everyone has moved to a more functional style of writing JavaScript Caitlin still insists on writing old-style imperative for loops which are super noisy. Get with the program Caitlin.

Dales team are on fire! Not literally but they did just knock out a really robust, fault tolerant data pipeline project in just shy of a week. Mind. Blown. Gold stars all round folks.

Finally we have Eda. Eda is like a magpie. She just loves kicking off projects in whatever shiny new technologies are at the top of Hacker News. The problem is she gets bored and distracted by other new things. Eventually she just moves on to another project. For pity sake Eda, sort it out.

Now these are interesting stories and I'm sure you can identify various people in these personas but there's a problem here. A few problems in fact which we will get to later. For now lets focus on one problem. All these stories are missing something. Context.

Context is to understanding like oxygen is to a fire. Without context it is impossible to fully understand any situation. Without context we are forced to make assumptions and judgements, we are more easily coerced into believing things about the situation and ultimately make wrong decisions.

So let me provide you a bit of context for the previous case studies.

Back to Alice - the dirty little hacker. Turns out Alice is working on a series of really rapid prototypes to help a client understand some of their internal requirements and discover what actually needs to be built. These prototypes are all throwaway and Alice knows that codifying assumptions through tests would be a pointless goal.

We're sorry Alice we were wrong about you.

Now Bob. Bob. Bob. Bob. Bobs nobel application of the "Boy Scout Rule" has really mucked things up for the support team. You see Bob had a goal of making code more concise but he ignored the fact that someone needs to maintain this code and so Bob set about code golfing perfectly readable, nicely structured code into a tangle of chained functional style calls. Bob made the support team cry.

Ah yes Caitlin. Going against the grain with coding style, using old style for loops in JavaScript when there is a perfectly good functional approach available. Well... Caitlin is right. You see Caitlin is working in a very performance sensitive part of the application and, well, old-style for loops are often much faster than their functional cousins. She used data to measure performance and came to this conclusion using her findings. Caitlin you used evidence based decision making instead of just falling in line. You rock.

Oh look it's Dales elite team of rockstar ninjas. Thanks for the robust data pipeline folks but I only wanted you to scrape a couple of web pages and extra some figures. In fact while I waited for you I wrote a little shell script using curl, sed and awk to do it for me. Your entire team just burnt nearly a week of funding due to over-engineering. Thanks for nothing Dales Team.

Finally Eda. Eda likes to remain current as it helps her make more informed decisions. So in her down time she works on hobby projects and code katas. She rarely finishes them because there is so much churn in the world of technology that she get distracted. Thats cool, she learnt stuff, she was under no obligation to actually release anything. Eda makes better decisions in work because of her tinkering and is well respected. Have a big fat pay rise Eda.

So you can see with more context we can actually alter the entire narrative of a story. What was once a bad thing becomes a good thing and vice-versa. Of course it may not. Eda could well have been one of those people that likes to start projects in work, complicate matters and then leave. But she wasn't.

And that is the point of this entire talk. With enough context we can make better decisions, with better decisions we can be more successful and start moving toward more betterness.

But it's not just context other things where at play such as bias and influence. So in order to achieve more betterness we need to understand these things as well.

First and foremost establish as much context as possible. Context is king after all. You can see how without context I was able to alter the narrative of a story. Know that there may always be more to it. Use discovery to improve understanding of the context. There is a reason why a great discovery session starts with as many diverse stakeholders as possible - everyone has a slightly different view of the overall context.

Don't walk into any situation with rigid assumptions, don't “solutionize" before you know enough. Listen.

I'm sure as I was reading out the case studies some of you had some reaction to the stories I was telling. Perhaps Alice and her lack of tests made you roll your eyes. Thats bias at play. Bias actually reinforced the narrative irrespective of the situational context.

We all have bias and bias exists without context. Bias affects your understanding of the context and so can adversely affect decision making. While we can't necessarily change our bias we can be aware of it. This helps us better understand our decision making.

In fact not being aware of a bias you may hold can lead to unintended steering of decision making.

One thing I did during the original case studies was, at least attempt to, influence your feelings for the person. I intentionally used negative words and tone to make you think the worst.

Influence may know about situational context but doesn't care. The loudest voice in the room may get their way but they may not be right. Influence can be either internal or external. That echo chamber of a Twitter stream that tells you how "OO is bad and functional programming is the one true way" can affect the decisions you make because of the influence it has over you. The thing is no-one wants to blog about daily maven usage when you get more social engagement and karma from writing about SBT internals.

And this brings me on to the team.

When I say team I mean a proper team not a group of people thrown together because the resourcing pipeline spat out a few available candidates. Some may have started out that way of course. I mean a group of people that have come together under a common cause. A team of people, not resources.

The great thing about a good team is that it normalises everything. Biases balance out, influences are moderated and everyone works together to form the truest view of the given problem context. The team establish context, the team make decisions.

A good team will do wonderful things. A bad team can lay the groundwork for years of pain. A good team will coalesce on an ideal. A bad team will bicker. A good team will make informed decisions quickly. A bad team will flounder. A bad team will cost you reputation and money. A good team will address problems upfront and openly. A bad team will let things fester until peer review time. A good team is diverse as it naturally allows for a broader range of thinking. A bad team is a monoculture.

So remember this.

Your team is your (work) family and as such you should love your team. Give a damn about the people and leave your ego at the door.

And of course I wasn't quoting some horrible Harry Potter erotic fanfic describing some carnal act between Harry Potter characters. Snape needed medical attention. Shame really.

The right context really changes the way a mind thinks.

More Betterness

Wed Sep 09 00:00:00 UTC 2015

Refactoring Towards Transducers

Tue Aug 04 00:00:00 UTC 2015

This is a quick post but hey at least I'm not talking about Elm. COMBOBREAKER!

Clojure 1.7 introduced a new, fairly large concept - transducers. Transducers are "composable algorithmic transformations" which means they are little isolated transformer functions. They aren't bound to any particular input or output and as such can be stacked or composed ad-hoc to make neat little pipelines for data processing.

This required a fairly large internal change in the way many collection functions in Clojure behaved. Now most of these functions now accept an arity of one less than they normally do. For example,

(def incrementalizer
  (map inc))

In this case incrementalizer is defined as a map function that will perform inc over whatever it gets passed. We didn't pass map a data structure to map over and so it returns a transducer. We can use comp to compose a bunch of these transducers together and run them across a collection using forms such as into, transduce, eduction and sequence.

As I was working on a new project recently I had just moved to Clojure 1.7 and so decided to take a stab at refactoring a few utility functions I had into a series of transducers. This post covers how I approached this refactor.

It's worth noting that this is purely an exercise in refactoring towards transducers. I am not suggesting that this increases performance, is easier to maintain or makes code any more readable. I'm not even suggesting this should be done at all as that depends on the context of your work.

The original approach looked like this,

(ns user
  (:require [clojure.java.io :as io]
            [clojure.string :as str]))
            
(defn- list-files [folder]
  (file-seq (io/file folder)))

(defn- is-xml? [file]
  (-> file
      (.getName)
      (str/split #"\.")
      (last)
      (= "xml")))
                         
(defn- list-xml-files [files]
  (filter (fn [file]
            (and (.isFile file)
                 (is-xml? file)))
          files))

(defn- list-xml-paths [folder]
  (map (fn [file] (.getAbsolutePath file))
       (list-xml-files (list-files folder))))
       
(def xml-paths 
  (list-xml-paths "./workspace/xml"))

A bit of scrappy code that I wrote interactively while doing some ad-hoc file processing. What we end up with is a list of absolute file paths to some XML files contained within a specified folder. In order to transform a folder path to a list of XML files it does a few things,

List the contents of the folder using the built-in file-seq function in Clojure
Remove everything in the list that isn't a file (such as pipes and directories)
Remove everything in the list without the .xml extension
Call .getAbsolutePath on the remainder of the list to get the full path as a string

Towards Transducers

Any place that we apply some sort of filtering, mapping, reduction, take etc. over a collection is a candidate for being converted to a transducer. In the previous examples we can identify a few points that can be rewritten as transducers,

The filter which actually has 2 checks
- is it a file?
- is it an xml file?
The map that converts a list of XML files into their absolute paths

If we rewrite these areas as transducers we come out with the following code,

(def xml-extension-only
  (filter (fn [file] (-> file
                         (.getName)
                         (str/split #"\.")
                         (last)
                         (= "xml")))))

(def files-only
  (filter (fn [file] (.isFile file))))

(def as-path
  (map (fn [item] (.getAbsolutePath item))))

you should be able to see these transducer definitions are just restructured, single arity versions of our original filter and map calls.

Now we need to turn these into a pipeline. We do this using the comp function,

(def xform 
  (comp files-only xml-extension-only as-path))

The use of xform as a name is common in transducer examples (short for transform) so I have retained the use here for consistency.

With our transducers composed into a transform pipeline we can apply it to our input,

 (defn- list-xml-paths [folder]
   (sequence xform (list-files folder))

I redefined the list-xml-paths function to call sequence over the list of files and apply our transducer pipeline.

The complete code for this example looks like this,

(ns user
  (:require [clojure.java.io :as io]
            [clojure.string :as str]))
            
(defn- list-files [folder]
  (file-seq (io/file folder)))

(def xml-extension-only
  (filter (fn [file] (-> file
                         (.getName)
                         (str/split #"\.")
                         (last)
                         (= "xml")))))

(def files-only
  (filter (fn [file] (.isFile file))))

(def as-path
  (map (fn [item] (.getAbsolutePath item))))

(def xform 
  (comp files-only xml-extension-only as-path))

(defn- list-xml-paths [folder]
  (sequence xform (list-files folder)))
  
(def xml-paths 
  (list-xml-paths "./workspace/xml"))

Deconstructing Your First Elm App

Sun Aug 02 00:00:00 UTC 2015

This article is similar to the Elm Architecture Tutorial so if you fancy a more official explanation you should check that out too.

One of things that has really impressed me in my research of Elm has been the simplicity of the architecture of an application. An Elm app is essentially just a reactive stream of actions that transition the model into various states and in turn these state changes cause the view to redraw. You don't mutate anything you just create a new state and the entire application redraws itself. Oh and it's fast too. This is thanks to the virtual DOM approach that has risen to fame through the likes of Elm and React.

Essentially Elm allows you to treat the UI as an Immediate Mode UI.

Deconstruction

What I want to do is demonstrate this simplicity by creating a very simple application firstly using the start-app package and then without using the start-app package. By using the start-app package initially we can see how an Elm app hangs together at a high level then, by removing the use of the package, we can dive a bit deeper into how our application is made reactive with some really simple functional patterns.

Using start-app

The start-app package abstracts away most of the boilerplate required to wire up a typical Elm application. It is designed to help people new to Elm get their first application up and running without to much upfront cognitive load. You just supply the StartApp.start a type containing

Your model that is used to store state
Your view that provides a visual representation of the current state, and,
Your update method that, through action signals generated from inside or outside your application, transitions your model from one state to another

If you want to follow along you need to have Elm installed. Have a look at the official site for instructions and various editor plugins for Elm.

So lets create our first application.

> mkdir elm-demo-start-app && cd elm-demo-start-app

> elm package install

Some new packages are needed. Here is the upgrade plan.

  Install:
    elm-lang/core 2.1.0

Do you approve of this plan? (y/n) y
Downloading elm-lang/core
Packages configured successfully!

This gives us the necessary manifests and caches that you get with any package managers in Elms case we get,

elm-package.json which declares all the metadata and dependencies our application needs
elm-stuff/ which acts as our package cache. You'll see the source of the dependencies we downloaded in the previous steps.

Then we want to add the extra dependencies that allow us to quickly build our Elm app,

> mkdir elm-demo-start-app && cd elm-demo-start-app

> elm package install

Some new packages are needed. Here is the upgrade plan.

  Install:
    elm-lang/core 2.1.0

Do you approve of this plan? (y/n) y
Downloading elm-lang/core
Packages configured successfully!

> elm package install evancz/elm-html 3.0.0

To install evancz/elm-html I would like to add the following
dependency to elm-package.json:

    "evancz/elm-html": "3.0.0 <= v < 4.0.0"

May I add that to elm-package.json for you? (y/n) y

Some new packages are needed. Here is the upgrade plan.

  Install:
    evancz/elm-html 3.0.0
    evancz/virtual-dom 1.2.3

Do you approve of this plan? (y/n) y
Downloading evancz/elm-html
Downloading evancz/virtual-dom
Packages configured successfully!

> elm package install evancz/start-app 1.0.0

To install evancz/start-app I would like to add the following
dependency to elm-package.json:

    "evancz/start-app": "1.0.0 <= v < 2.0.0"

May I add that to elm-package.json for you? (y/n) y

Some new packages are needed. Here is the upgrade plan.

  Install:
    evancz/start-app 1.0.1

Do you approve of this plan? (y/n) y
Downloading evancz/start-app
Packages configured successfully!

So far, via the most polite package manager I've ever encountered, we have installed

evancz/elm-html which allows us to work and build HTML and associated elements attributes, events and CSS styling
evancz/virtual-dom is installed as dependency of elm-html and provides the virtual DOM implementation that Elm uses.
evancz/start-app a small library for making writing standard Elm apps with less upfront wiring.

TO THE CODE!!

The app is going to be really basic, and I really do mean basic. Just enough to demonstrate the core components of the Elm architecture - the model for holding state, the view for visualising app state and the update function for transitioning our state.

The app will have 2 buttons - "Mark" and "Reset". When you click "Mark" the app will append a block (█) character to represent that event to the event log and increment the number of events logged. When you click "Reset" the app will clear the event log and zero the number of events logged. Contrived? Yes of course. But acceptably so.

Starting Point

So lets create an EventLog.elm file. This will be our one and only code file for the application. We'll start by declaring the module and importing the necessary packages,

module EventLog where

import Html            exposing (..)
import Html.Events     exposing (..)
import Html.Attributes exposing (..)
import Signal          exposing (..)
import StartApp

I've just imported a lot more than we will need for this app but feel free to curate your imports a bit better if you so desire.

Now we will start from the bottom up and declare our main function. This is the starting point for all Elm apps. It will usually return a Signal of some type that can be used by the Elm infrastructure to perform the necessary (re)rendering of your application. You can think of a Signal as a typed stream that emits items as and when they are generated.

As we are using the start-app package ours will look like this,

main : Signal Html
main =
  StartApp.start
    { model = initialModel,
      view = view,
      update = update }

I have included the type signature in here for good measure. Our main simply calls into StartApp.start passing in our 3 main components. Of course we still need to define these so lets go ahead and do this.

Model

Our model will define the entire state of our application. In our case this won't be very big at all, in fact it could literally just be a counter the we increment but for the sake of this experiment we will make it slightly more verbose,

type alias Model =
  { events: List String }

initialModel : Model
initialModel =
  { events = [] }

I defined a type alias because it avoids the mess of having to declare the type as { events: List String } in every type signature that includes it. Then I defined our initialModel using the alias. The initialModel, as the name suggests, is our starting state for our application. An empty model.

View

So now we have our state we can create a visual representation of that state by creating our view,

view : Address Action -> Model -> Html
view address model =
  div []
    [ button [ onClick address (Mark "?") ] 
        [ text "Mark" ],
      button [ onClick address Reset ] 
        [ text "Reset" ],
      h2 [] 
        [ text (model.events |> List.length |> toString), text " Events" ],
      div [] 
        (List.map (\t -> text t) model.events)]

I'd typically break this into smaller functions to keep it a bit cleaner but for explanation purposes this is good enough. Our view takes an address and model. The address is passed in from the work done by start-app and provides a mailbox to send messages our actions to in order to update the model. The model in this case is the current state of the entire application.

With these key bits of data we construct a primitive HTML representation.

We define buttons that have onClick handlers. These handlers simply send actions (we define this type later) to the supplied address.
We define an h2 element that counts the number of events in our event list and displays it
Finally we define a div that converts our events list into HTML elements

Actions

Lets go back to this address and actions. As mentioned the start-app package creates an internal mailbox and address that you can send things to which will get passed into our view function. When we send stuff to the address this will ultimately trigger a call to the apps update function. As you can see from the type signature of view the Address is of type Address Action. Action is a type that we will define so lets do that and talk about it after.

type Action = Mark String
            | Reset

If you think back to the UI we had 2 buttons - Mark and Reset. What the Action type does is create a representation of these UI interactions that can be used within our app to signal intent. Types like the Action type - the ones that use | - are called union types. In some ways you can think of them as enums that can accept data as well. By defining this Action type we explicitly declare all the actions that will change the state within our application. No just can but must. That's predictable, thats wonderful and lovely and makes me feel all warm inside.

So, yes sorry I got all excited, the buttons we defined in our view method, each, sends one of these types to the supplied mailbox address. So lets define what happens when we do this,

Update

Right. We have a model, a view and a way to signal changes in state via actions. Now we need to actually manage those state changes. This is where the final piece of the puzzle falls into place,

update : Action -> Model -> Model
update action log =
  case action of
    Mark date ->
      { log | events <- log.events ++ [date] }
    Reset ->
      { log | events <- [] }

The update function is just one big case statement. It gets passed an action and the current state of our application and depending on the action we return a new state for our application. This generation of new state causes our application to be re-rendered and the glorious cycle begin again.

For completeness sake here is the entire EventLog.elm

module EventLog where

import Html            exposing (..)
import Html.Events     exposing (..)
import Html.Attributes exposing (..)
import Signal          exposing (..)
import StartApp

type alias Model =
  { events: List String }

initialModel : Model
initialModel =
  { events = [] }

type Action = Mark String
            | Reset

update : Action -> Model -> Model
update action log =
  case action of
    Mark date ->
      { log | events <- log.events ++ [date] }
    Reset ->
      { log | events <- [] }

view : Address Action -> Model -> Html
view address model =
  div []
    [ button [ onClick address (Mark "?") ] [ text "Mark" ],
      button [ onClick address Reset ] [ text "Reset" ],
      h2 [] [ text (model.events |> List.length |> toString), text " Events" ],
      div [] (List.map (\t -> text t) model.events)
    ]

main : Signal Html
main =
  StartApp.start
    { model = initialModel,
      view = view,
      update = update }

Running the App

Elm comes with some lovely tools to get your first app up and running without having to worry about the usually front end infrastructure such as grunt, gulp, bower, npm and so on and so forth. One of these is reactor

> elm reactor

Elm Reactor 0.3.2 (Elm Platform 0.15.1)
Listening on http://0.0.0.0:8000/

Running elm reactor starts up a little file server (typically on port 8000) and if you navigate to http://localhost:8000 you will see the project files listed. By clicking EventLog.elm reactor will compile your Elm code to JavaScript and inject it into an ephemeral HTML document so you can see and play with our utterly useless creation.

Wrapping Up start-app

So in summary

The main function starts our app using StartApp.start which we pass our three main components
The model represents the current state of our entire application
The view provides a visual representation of our current application state and also generates action signals from user interaction that trigger an update in our application
The update function takes these actions and creates a new state model for our application which triggers a re-rendering of our view

This is a deceptively simple approach that avoids one of the main pain points in application development - mutable state. Having to manage mutable state introduces a lot of complexity and unpredictability - two things that should never really mix.

Removing start-app

So the next question is "what does start-app do?". As I've demonstrated start-app has provided the glue for realising our application - we just provided the components. Knowing how these things glue together really helps with understanding Elm and its additional features such as signals and ports.

The Mailbox

As we've seen in the previous implementation our view function gets passed an address something that we didn't create ourselves. Internally start-app creates a mailbox - a concept in Elm that provides a typed address that we can send messages to and a signal that emits a value when the address receives a message. So you can see that when we send an action message to the address via the onClick handler in our view we can then trigger a state update.

Without the start-app we need to create the mailbox ourselves,

actions: Mailbox Action
actions =
  mailbox Reset

Our actions mailbox has a type of Action and when we create our mailbox we supply it an example of our Action type (in this case Reset). Often you'll see action types defined with a NoOp and that is often used as an example. Either way he have our mailbox.

Reactive Model

Next up we need to take our initial state and turn it into a reactive model. So that when an action message is received it will cause the update method to alter our state. Now this part is, IMHO, just the cleverest little pattern. The reactive model is just a fold over the signal that is a property of our mailbox starting with our initialState

model: Signal Model
model =
  foldp update initialModel actions.signal

OK technically it's a foldp which is a special signal variant of fold that can yield intermediate results (as an when actions are generated). But thats it. Starting with our initial data when actions occur call the update method - the result of which is yielded by foldp. We have our reactive model.

Mapping Main

Finally we need to replace our main method and wire up all these parts to create our working application.

main : Signal Html
main =
  map (view actions.address) model

Now main becomes a map over the view using our model (the one we made reactive in the last step). As view takes both an address and a model so we create a partially applied or curried function of (view actions.address) that takes the address from our mailbox. So when the model updates the map will cause the view function to be called with the new state which will build our HTML representation and yield a signal value with that HTML representation into the core of Elms rendering engine. Once again the map function (from the Signal package) yields intermediate responses.

And rest...

So that covers the basic architecture of a typical Elm application. You can see how start-app removes some of the extra steps required to wire up an Elm application and would certainly lower the barrier of entry for people new to Elm. That said there are times you want access to the Mailbox (such as when trying to merge in external signals from ports) and start-app hides the mailbox it creates on your behalf.

The underlying principles of Elm are surprisingly simple yet very powerful. They aren't niche concepts either. Fundamentally they are common functional patterns which means, with a modicum of functional thinking, that application design is less made complex as we have to incur less cost in understanding concepts specific to Elm.

Zombie Dice Score Card in Elm

Wed Jul 29 00:00:00 UTC 2015

As promised yesterday evening I spent some time re-writing my Zombie Dice Score Card, originally written in ClojureScript and Reagent, in Elm.

If you just want to poke around the code then the source for each implementation is available on Github,

If you want a bit of an experience report then read on.

This work was done after spending an evening with some Elm screencasts and some internet research I cannot promise this is good or correct code and I'll welcome any feedback via Twitter

Developing the Elm version of the app took the best part of 3 or 4 hours once I discovered how to integrate with JavaScript via ports. All in all I'd say there is about 6 or 7 hours work to craft my first Elm app with only a couple of hours introduction to the language. This should be seen as a positive reflection on Elm and not me.

Types

My previous experience with type focused languages was a few years of general purpose Scala development and a brief foray into Haskell which ended in mild frustration. I'm certainly more comfortable in dynamic languages and if I was ever bored enough to be drawn into a "dynamic vs static" debate I'd probably side on the dynamic side. With that said picking apart my ClojureScript implementation and recreating it in Elm has really made me appreciate its typed nature. The ability to look at our defined types e.g.

type alias Player =
  { id       : Int,
    name     : String,
    gamesWon : Int,
    score    : Int }

type alias ScoreBoard =
  { players : List Player,
    uid     : Int }

This explicit declaration really calls out stuff that, in ClojureScript (or my ClojureScript code at least), is implied and typically scattered over the place. For example, in the code below I'm just pulling out keys from a map :id, :games-won etc. because I know these things should exist.

 (defn- finish-game []
   (let [players (vals @scoreboard)
         winner (reduce #(if (> (:score %1) (:score %2)) %1 %2) players)]
     (swap! scoreboard update-in [(:id winner) :games-won] inc)
     (reset-scores)))

Last night when I revisited this code I had forgotten what my model looked like and I had to figure it out by scanning all my code for when things are set and read from my data structure representing my state. I'm sure I could structure my ClojureScript better, perhaps use something like core.typed or Prismatic Schema, but the onus is on the developer. Of course this is not a new and startling revelation this is just one of the many difference in dynamic/static languages. So what does Elm do to make this different? Well, for one, it uses gradual typing. I can declare a function like this,

newPlayer id name =
  { id       = id,
    name     = name,
    gamesWon = 0,
    score    = 0 }

This will work fine and the Elm compiler will infer the types and shout at me if I'm passing ambiguous types around. However I can solidify my intentions by adding a type signature for the function,

newPlayer: Int -> String -> Player

When I initially started dipping into Elm I thought these signatures would be an after thought for me. I'd only add them in for completeness sake. I was wrong. They are extremely useful and helped massively as I designed my implementation. They also improve the error messages. Scala has this sort of type inference as well but I almost always left it out unless required. Perhaps I've got wiser in my old age or maybe Elm has made the type system more accessible.

Error Messages

When something goes wrong in Clojure or ClojureScript the resulting stacktraces can be next to useless for figuring out what is actually wrong. No amount of formatting and colouring improves this, sometimes they are just confusing. Error messages in Elm are the complete opposite.

-- TYPE MISMATCH  ZombieDice.elm

The type annotation for `playerEntry` does not match its definition.

98| playerEntry: Address Action -> Player -> Html
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Looks like a record is missing the field `uid`

As I infer the type of values flowing through your program, I see a conflict
between these two types:

    Player

    { a | score : Int, uid : Int, gamesWon : Int, name : String } Use --force to continue.

Just look at that message. It's telling me exactly what is wrong and I knew how to fix it once I'd read this message. Most Elm messages are like this although some of the type related ones did make me scratch my head until I wrapped my head around them a bit more.

HTML

Hiccup style syntax in ClojureScript that we use for describing HTML structure is lovely.

 [:div.row
  [:div.col.span_3.name
   [:h2 (:name player)]
   (for [win (range (:games-won player))]
     [:img { :src "images/gold-brain.png" :height 20}]) ]
  [:div.col.span_3
   [:image.brain { :src "images/brain.png"
                   :on-click #(update-score (:id player)) }]
   [:span.score [:small " x "] (:score player)]]]

With paredit enabled creating this sort of structure is an absolute breeze and rather easy to comprehend after it's been written. The same, i'm afraid, can not be said for Elm,

  div
    [ class "row" ]
    [ div
        [ class "col span_3 name" ]
        [ h2
            []
            [ text player.name ],
          span
            []
            (List.map (always wonGame) [1..(player.gamesWon)])],
      div
        [ class "col span_3" ]
        [ img
            [ class "brain",
              src   "images/brain.png",
              onClick address (Inc player.id) ]
            [],
          span
            [ class "score" ]
            [ small
                []
                [ text "x" ],
              span
                []
                [text (toString player.score)]  ] ] ]

This is formatted, I think, using the recommended formatting advice and while it is relatively easy to read and get to the place you want to make changes it caused me a lot pain creating it and refactoring it as I added new features. I almost always misjudged bracket placement and forgot commas resulting in errors. Perhaps it's the fact paredit gives me so much power and ClojureScript doesn't use commas (they're treated as whitespace) but this was painful for me to create.

It's just one big fold

The architecture of an Elm app is always the same and it's really rather clever. However there is a bit of initial setup to get everything working. To this end you can use the start-app package that removes a lot of the initial boilerplate (there's not really that much boilerplate in reality). I didn't use this package as I wanted to really understand how everything was wired together.

This led to my big revelation. The root of an Elm app is just a map our reactive model that returns the updated view as and when our state changes. The reactive model part is just a fold over the application data which gets updated actions/signals happen that cause the state to transition to another state.

The entire app updates when something changes the state.

No management of discrete little parts and internal state - just rebuild the world. This is the same principle behind React and Reagent (a ClojureScript React wrapper) but in React the mechanisms are somewhat hidden. In Elm it's right there. You can compose different signals, à la Reactive Extensions Observables, to make your app behave in different ways. For example,

model : Signal Model
model =
  let
    allActions = mergeMany
      [ actions.signal,
        externalActions ]
  in
    foldp update initialModel allActions
    
externalActions: Signal Action
externalActions =
  mergeMany
    [ (Add) <~ addPlayer ]

In this example from the app I merge the actions signal that comes from the apps main Mailbox (this is the one that streams all the internal Actions to update the model) and external signals (these are the ones that are generated by JavaScript outside of the Elm app). externalActions does some manipulation of it's incoming data to create a signal of type Action that can be merged with our core Action signal.

While I have likely explained this in the worst possible way this approach makes everything wonderfully composable and straightforward. As soon I saw how Elm approached reactive application development I was just blown away by its simplicity.

Conclusion

I like Elm. Go try it. I can't really say one implementation of my little app is better than the other each has it's good and bad points. Besides I haven't spent enough time with Elm to evaluate it properly yet. All I know is that I've taken to it better than other languages of its ilk.

I'm not sure why it's type system feels more accessible to me compared to Scala or Haskell but I've found that the barrier of entry for Elm is much less than either of those languages. There seems to be, compared to Scala at least, much less concepts to wrap your brain around while still remaining extremely powerful.

Finally I recommend you check out this talk by Evan Czaplicki (creator of Elm) around the design decisions that help make Elm more accessible.

window.prompt() in Elm

Tue Jul 28 00:00:00 UTC 2015

I say "standard stuff" a few times in this article. When I say that I assume, like me, you've dipped your toes into Elm. If that is not the case and you want to change that situation then I recommend go give Pragmatic Studios your money. I was in no way paid or influenced to say that BTW.

Having spent last night watching the Pragmatic Studios Elm: Building Reactive Web Apps I am now a fully certified expert in Elm and so it's time to start writing articles about it. Of course this is nonsense but some of my adventures have inspired me to write an article or two that may be useful to others (and my future self).

I recently documented a little hobby project I worked on for keeping score on games of Zombie Dice. The app was written in ClojureScript and after I had wrapped my head around Elm I decided to see what the same app would look like in Elm. This article isn't about that, but rather a small part of it that I found quite challenging.

Unlike ClojureScript (and Clojure) whose philosophy is to embrace the underlying host environment Elm abstracts it away. It's not gone completely it's just not as prevalent or easy to access. In my Zombie Dice Score Card app adding new players uses the standard JavaScript host function window.prompt to capture the name and I wanted to replicate this functionality in my Elm implementation.

Basic Implementation

My first pass at this resulted in a working implementation (well not my actual first pass it was many, many passes before I even figured out what the hell I was doing). While it wasn't going to be practical for my actual needs it did form the basis of something useful that I could build upon.

The key is to using ports. A port acts as a bridge between Elm and JavaScript. They either go in to Elm or out of Elm. You declare a port in Elm like this,

port suppliedNames : Signal String

This is port is an innie. You can tell it is an innie because you supply no actual definition. Instead we can send things to it from the JavaScript side.

app.ports.suppliedNames.send("James");

Assuming, in our HTML file, our Elm app is created and assigned to a variable app we will have this ports object that lists all the ports we expose from our app. This will have our suppliedNames port which will have a method of send that we can use to signal values through that port.

One other thing is that these innies must be given an initial value when the app is constructed or you'll get an exception. We can do it like this,

var app = Elm.fullscreen(Elm.Confirm, { suppliedNames: "" });

So when we boot our app we pass in an initial value.

Lets look at another port,

port totalCapacity : Signal String
port totalCapacity = map toString somethingElse

This one is an outtie. This is used to signal things out of our Elm app into JavaScript. In this case it sends out the sum of some data structure as and when it changes. We can listen to these signals in JavaScript like this,

app.ports.totalCapacity.subscribe(function(x) { 
  console.log(x) 
});

Outtie ports have a subscribe method generated for them that takes a function accepting the payload. In our case we simply log it.

Now onto our specific use case. My needs are slightly different to the samples because while they broadcast data out of the Elm side I just want some sort of trigger to say "go ahead and open a prompt". This is 100% impure in that it is only used for side effects and I've found that with Elm being a rather pure language this sort of requirement feels kind of awkward. Thats not a criticism, just an observation. You should expect bad things to feel awkward, it helps you minimise them.

The JavaScript/HTML side of our application is fairly unsurprising (assuming you've not skipped the last few paragraphs).

<!DOCTYPE HTML>
<html>
  <head>
    <meta charset="UTF-8">
  	  <script type="text/javascript" src="confirm.js"></script>
  </head>
  <body>
  </body>
  <script type="text/javascript">
    var app = Elm.fullscreen(Elm.Confirm, { suppliedNames: "" });
    app.ports.prompt.subscribe(function(){
        var value = window.prompt("Players name?");
        app.ports.suppliedNames.send(value);
    });
  </script>
</html>

Here we,

boot our app passing in a blank value for suppliedNames
subscribe to a prompt port which will display the window.prompt
send the captured value from window.prompt back into our app

The Elm side of things is where the real stuff happens,

module Confirm where

import Html        exposing (..)
import Html.Events exposing (..)
import Signal      exposing (..)

-- SIGNALS 
type Action = NoOp | Prompt

actions : Signal.Mailbox Action
actions =
    Signal.mailbox NoOp

-- PORTS --
port suppliedNames : Signal String

port confirm : Signal ()
port confirm =
  actions.signal
    |> filter (\s -> s == Prompt) NoOp
    |> map (always ())

-- VIEWS --
view name =
  div []
    [ text name,
      button [ onClick actions.address Prompt ]
        [ text "Set Name"]]

-- APP -
main : Signal Html
main = Signal.map view suppliedNames

Given there is a fair few things happening here lets have a look at the interesting bits.

First of all we set up the various operations our application will perform and create a Mailbox that can be used to send actions to. If you've ever tinkered with Elm this should be pretty common,

type Action = NoOp | Prompt

actions : Signal.Mailbox Action
actions =
    Signal.mailbox NoOp

The Prompt action is the one we want to use to trigger our window.prompt call via a port. Next we declare our actual ports,

port suppliedNames : Signal String

port prompt : Signal ()
port prompt =
  actions.signal
    |> filter (\s -> s == Prompt) NoOp
    |> map (always ())

suppliedNames is our innie. This will receive names we've created later on. prompt is our outtie. What we do is

take the signal from our Mailbox
filter everything except the Prompt actions
map these values an always return a unit value

Why unit? Well we don't care about the actual value we just want to reach out to the JavaScript when we get a Prompt action through our mailbox. Any value is meaningless so lets just go for the most meaningless one we can find.

Then in our view we have a button that sends a Prompt action to our Mailbox,

button [ onClick actions.address Prompt ]

Again this should be fairly unsurprising to the Elm-ites (or whatever the collective term is) reading this.

Finally we have our main that wires it all together,

main = Signal.map view suppliedNames

Again - standard stuff.

Notes

As the last title indicated this is just a basic implementation. It requires a bit more work to integrate into an application that requires a more complicated model but its a good starting point. I know this because I've already begun extending it to create the Elm version of my Zombie Dice Score Card which I'll dissect in another article soon.

Now I'm not saying this is idiomatic Elm, I'm not even saying this is a recommended way to do things but it works and "feels" good enough to me. If you want to comment/fix/critique/whatever my work then you can get me on Twitter and let fly the dogs of conversation.

Zombie Dice Score Card

Mon Jul 20 00:00:00 UTC 2015

Zombie Dice Score Card with Reagent

Mon Jul 20 00:00:00 UTC 2015

Over the weekend I built a Zombie Dice Score Card. I built it because I keep forgetting to get pen and paper for keeping score but mostly because I wanted to build something simple in ClojureScript again.

The app is pretty basic, no server side interaction, no local storage interaction and only a few basic actions/features. You simply,

Add players
Play zombie dice as usual, rewarding points by clicking the necessary brain on the screen.
When someone reaches 13 brains you'll be told to finish the round (all remaining players yet to go on this round have a final chance at stealing the win)
Hitting 'Finish Game' will reward the person with the highest number of brains a gold brain and the next game begins.
At any time you can reset the current game scores or clear the entire board.

For the curious the stack used is as follows,

ClojureScript
Tenzing - Leiningen template for creating Boot powered client side only ClojureScript applications.
Boot - The new build system on the Clojure(Script) block offering a more programatic approach to build systems (i.e more like Rake than Maven)
Reagent - A simple ClojureScript wrapper around React.
cljs-uuid-utils - ClojureScript library for generating and working with UUID's

Tenzing

This is a nice little template for prototyping with ClojureScript or in situations where you can integrate with external services for your server side needs. It's a conditional template that lets you add technologies that you think you'll need. For example I generated the Zombie Dice Score Card project with,

lein new tenzing zombie-dice +sass +reagent +divshot

This template gives me an app with SASS, Reagent and Divshot configuration and all the necessary build pipeline stuff set up to compile CLJS and SASS when files change. I didn't use Divshot this time around but it's nice to know the option was there.

On another note I did find it a bit weird using Leiningen to generate a Boot application but I guess most Clojure(Script) developers are going to have Leiningen available and Boot doesn't provide templating support.

Boot

I didn't need to dive into Boot too much for this tiny project but I do like what I see. The way tasks are defined in Clojure and composed into pipelines using standard Clojure idioms makes creating build pipelines really rather nice.

(deftask build []
  (comp (speak)
        (cljs)
        (sass :output-dir "css")))

(deftask run []
  (comp (serve)
        (watch)
        (cljs-repl)
        (reload)
        (build)))

You see run composes a pipeline of other tasks for serving the application, watching for filesystem changes, connecting to a ClojureScript REPL and reloading the application. It then composes another defined task for building the ClojureScript and SASS.

Reagent

Aside from a name change Reagent (formerly Cloact) hasn't really change that much since I talked about in January 2014. Sure they've added some new features and refined the API a bit but its approach is still the same. It provides simple approach to using React in a ClojureScript app offering us hiccup style syntax for declaring components along side atoms for state management.

I've never really looked into the other React wrappers so I can't speak to whether Reagent is the best but I can say it's certainly been a pleasure to use on small scale apps.

Source

The source for the app is available, as always, on Github and I welcome any feedback or advice you may have.

Plans

I think I'm going to write this in Elm just to see what all the fuss is about. Maybe add some local storage for saving historical games and players. Truth be told I'll probably never touch it again but a man can dream.

Footnote

Have you ever played Zombie Dice? If not you should give it a go it's a simple, quick dice based game that you can explain in about 3 minutes and have a full game done in about 15 or 20 minutes. The aim is simple, try and collect/eat 13 brains by rolling dice over a series of rounds. Once someone hits 13 brains everyone finishes that round and the person that has the highest score wins that game. Typically you play best of 3 or 5 games.

Meteor: An Introduction by an Expert Beginner

Wed Jul 01 00:00:00 UTC 2015

Isolation

Wed Jun 24 00:00:00 UTC 2015

One morning you woke up and things were different. Not hugely different - just ever so slightly. Almost imperceptibly so. You didn't really notice it at first but eventually you felt that something seemed "off". You had got up, had breakfast, brushed your teeth, cycled to work, showered, made a coffee and sat down at your desk and you just sensed it.

Do you remember when you were a kid and you lost one of your favourite toys but found it again months later after having given up on the search. Did you ever get that weird sensation that it was much smaller, maybe less detailed than you remember? Well you're getting that feeling now, the space around you just "feels" smaller.

After a while you notice it. A wall. That wall wasn't there before - you are sure of it. It doesn't actually look new; in fact it doesn't even look out of place but you're positive it wasn't there before.

The next week, while going to another floor in the office to visit a work colleague, you notice another strange thing. One of the office doors isn't there anymore. As far as you recall there used to be another office between those two doors on the far wall, you'd had a meeting there once and you would swear to it. Other people don't seem bothered - "it looks the same to me" they say - and it's not as if there is evidence of some fresh plastering in the space. It just seems, well, smaller.

Over the next few months to a year little things like this happen. You notice, or at least feel, that things are getting smaller. There is less space to move around and even more widely there are less people occupying the space. Even places you've never been before feel significantly more constrained. There are people you haven't spoken in a while, you don't miss these people immediately because they aren't a really part of your core pack of people you see on a regular basis, but you did sense them missing.

On the moments you take to reflect on your life you notice just how small everything is getting. How compact, how claustrophobic everything is. Basic math tells you that there must be more space between these walls but there are no doors.

One day you wake up and go to your office but looking around you're not even sure how you really got there. There are no other doors just the one door from your house to your office.

You feel alone.

Isolation can happen for many reasons both internal and external. The reasons can be anything from mental state, actions to something as simple as an inflexible and strongly held opinion.

Individuals become isolated a team does not, not as readily at least. Teams, genuine teams of people that care about each other and the work they do, create an internal supportive culture. Individuals problems can be diminished through collective support. Toxic attitudes can be identified and a resolution can be reached.

In the same way a toxic attitude does not thrive in a strong team, a good team cannot thrive in an unsupportive or constrained organisational culture be it in a company or community. We may talk about how culture is important but I don't think we all consider what that means. Being inclusive and supportive requires real effort at all levels of an organisation.

Road Trippin'

Mon Jun 15 00:00:00 UTC 2015

This is a ramble. There is little originality in this post. It's quite long. There are some pictures. I'm OK with all these things.

Projects are a bit like road trips ¹. You know where you want to go and roughly how you think you'll get there, you have your team of people joining you on the journey and no amount of planning is going to ensure you get to where you want to be when you want to be there. That's just happenstance.

GPS

If projects are road trips then the GPS is your plan. Before you start off you plonk in your destination and after some internal calculations you get an approximate estimated time of arrival. It knows this because it assumes it knows what route you'll take and it knows a little bit about the roads that make up that route. From this data it is able to extrapolate an estimated time of arrival and total milage.

Most projects still have some element of an upfront plan. It may be very lightweight, just a figure like our GPSs ETA or it may be more detailed like the GPSs routing. Something upfront is useful for budgeting (or the approval of) and determining peoples availability for the project ².

The GPS and your project plan are both snapshots of the current state of play based on the knowledge they currently have. That massive 2 year Gantt chart that goes right down to the day level that you spent weeks on at the start of the project is immediately useless the day the project starts. The fact it may sometimes appear to be correct is purely by chance. Without taking current and historical data into account and updating the plan your information is lying to you. Having a flexible mechanism for feeding this real time information into your plan quickly is vital - an A1 dead tree Gantt chart mounted on your teams wall should be set on fire and the ashes scattered to the wind.

The plan itself is NOT a goal - it is a guess that gets better the closer to the real result you get.

I repeat the plan itself is NOT a goal.

Roadworks and accidents

There are some things a GPS system can't predict namely short term roadworks and accidents. Many systems incorporate some sort of up to date information about these things but not all. The minute you encounter a tailback from an accident or some other unpredictable slowdown in traffic the ETA starts slipping as the GPS uses the incoming data to readjust.

To reiterate - any form of project plan has to be able to use all available data. You can't simply assume the plan you came up with at your most ignorant is still holding true. The quicker you can adjust the plan the quicker you can notice trends and react accordingly. Overly detailed plans are usually both wrong and out of date and in that state the only thing they are good for is finger pointing and blame laying - two things that will ruin a teams morale and further damage the project.

Don't Miss the Boat

Of course the end destination (or even various destinations on your trip) may be time dependant. Perhaps if you have to catch a ferry. They may also be cost dependant too of course. Maybe you're a band of cash strapped musicians with £30 left to the next gig to cover food and petrol. Who knows, this is your fantasy - live it.

Anyway the GPS's ETA is still not a goal. Sure it is an indicator wether you'll meet your goal (or your ferry) on time but you can't blame the GPS for having an ETA that means you'll miss that boat.

In fact you can't even go blaming the manufacturers of the GPS - they developed an algorithm that takes all the information it has and, assuming that information is correct at time of calculation, tells you how long it will take. Things change. Understand that. We just have to do our best to ensure we get there when we need to by making sure we think ahead.

The same is true for projects. We'll not dive into why FPFT (Fixed Price and Fixed Time) projects are just horrible incubators for bad software and toxic environments. Instead lets focus on Fixed Price or Fixed Time. Many projects need to go live at a certain date. Incoming legislation may require it. The marketing department may have already made public announcements. Many projects can only get a certain amount of money because red tape demands it or a customers accounting practises are apply practises that don't fit well in the IT sector. The reasons exist for fixing some aspect of the project and they are many.

We need to ensure we can manoeuvre within these constraints. When something slips, and it will happen, we need to better understand how we can deal with this instead of trying manage expectations or deferring the truth until its too late.

It still amazes me that projects will spend longer on a piece of relative fiction like a detailed plan upfront yet pretty much skip any honest talk about what happens when the proverbial hits the fan and how to prevent such things.

Remember our GPS and its ETA? Well a plan that can be updated quickly and reflect the current state of play quicker. Make it visible. Like that little ETA on our GPS display that everyone can see. At any point in time, based on what we know, anyone can see if we are slipping and we can work out how to deal with a small slippage rather than waiting until near the end and realising you've missed the boat by a 1/2 a day.

Come See "The Largest Toenail in the World"

Road trips can be dull, even with the sweetest tunes and funnest friends, you'll realise that other peoples music sucks and everyone else has fallen asleep. You need to break up the monotony and you can do this by stopping at tourist spots. Get out, stretch the legs, change the scenery, have a nice long wee. Whatever.

Of course the more stops you have the more time it will take to reach the end goal. It may also take you out of your way and eat more fuel.

But you build this in. You can use the GPS to add via's and give you a more accurate view of how long it will take. You also add contingency. You leave the house a bit earlier to give you some breathing room in case anything or everything goes wrong. You agree that everyone is willing to miss "The Largest Toenail in the World" exhibition if it means shaving off some time to make that ferry or save some petrol. You check that tyre pressure and that oil to make sure the car wont suddenly explode.

Projects are no different except instead of visiting tourist spots we're talking about features. We discover what features people really need. We discover what people want but not need. We accept that should we risk missing our deadline or running out of cash we can drop non-essential features.

We de-risk scary parts of projects by addressing them early or spiking out our understanding of things we don't know yet or are afraid of. We need to question why we have certain features. Who requested that Oculus Rift enabled login screen or gesture based Rich Text Editor? Better yet don't question them. Come to a set of features by asking people who will actually use the system. Them beautiful users. Use them to build what users need. Build prototypes of features ahead of time, through the course of the project, to understand user needs rather than the vision of some middle manager who's never going to actually use the stuff they demand.

Most importantly communicate. Talk with your team. Talk with your customer. Be honest, be candid. Do it early, do it often. Small change, often, is better than monumental change right before a customer has to walk into a board of Directors and explain why everything is falling apart. People are going to be more open to change if they feel comfortable and trust you.

The more we know the better we deliver (though we may not deliver more or faster).

Easy Street

Not all parts of a road trip are fraught with risk like some terrible Mad Max B-Movie clone, some parts you could do blindfolded (not actually blindfolded mind you). Especially the start.

Getting out of the city, the one you where born and raised, is a piece of cake. You know those roads like the back of your hand, you don't even need the GPS at this point. Further on, you may have previously visited a village you're passing through and remember some local tips for avoiding that famous village market rush. You may find that you even shave some time off the journey.

You get this on projects as well. There will be points in the project that you or your team know so well through experience in a given problem or business domain.

It's a double edged sword though.

If you don't recognise this productivity boost is due to previous experience you may feel you're flying through the project smoothly when you may well be in the eye of a storm. If you can identify this as the cause of your new found productivity you can use this opportunity to reduce risk and further understanding. Try spiking out some riskier areas of the project. Imagine this like taking a new route on the way out of your city - it could lead to an even shorter journey. But it may not. At least now you know. You're keeping the risk low though as you are in familiar territory and can get back on track easily.

Take advantage and learn.

Flat Tyres and Engine Trouble

Cars - complex metal monsters rolling around on pressurised cushions of air and rubber. No matter how well you look after your car chances are something will go wrong with it. It usually something that can be fixed with a tyre or a fuse or a can of Radweld or something similar. You could call roadside assistance and wait for a few hours but you'd be going nowhere that whole time. But if you or someone your with knows how to change a tyre or replace a fuse you're going to get moving much faster. You may need to come revisit the repair later on but if a quick fix gets you moving (without compromising your safety) then you're going to make much better time overall.

Having a broad level of skill on a project team ensures that downtime for minor problems is minimised. I'm specifically talking about infrastructure and ops but the analogy can span design, testing and project management. The most productive teams I've been involved with have been the ones that can manage their own deployments in a controlled manner. They may have needed help getting the infrastructure up and running with a dedicated Ops team and have occasional reviews with Ops specialists to ensure everything is ship shape but all in all they were self sufficient for the most part.

Siloing specialisms may make sense from an organisational perspective but they add additional friction when it comes to the reason that organisation exists - project delivery. Cross functional teams should be nurtured.

The Town of Lost Souls

Sometimes your GPS is just not right. Data for out-of-the-way towns can be out of date and it can get you lost. If we're willing to accept stereotypes here, which I will, you're not allowed to get lost and if you do get lost you must not tell anyone. Winding your window down and asking the sweet looking old man how to get to Babbling Brook Lake, even though he so obviously knows exactly how to get there is simply not allowed. Instead, with absolute certainty, you know how to get there you make another lap of the town because you think you saw a sign - burning more daylight and fuel in the process. Eventually you admit you need help, swallow that pride and ask the next person who gives you the best instructions you've ever heard.

Communication on projects is essential. I'm not talking about awkward morning stand ups were people pretend everything is OK and waffle about stuff that they may or may have done when in actual fact they haven't even turned their computer on in days because they are scared of the code that refuses to work. I mean open communication in an environment that people feel safe and willing to admit they have problems. Admit you have a problem, allow others to admit the same. Then you can apply teamwork ointment and work through the problem together.

When people are in an environment that makes them cover up their mistakes and issues you end up with a bigger mess. When people are too full of pride or too cocky to admit they are having issues the team is in trouble.

Support and empathy go a long way to making people more productive.

Sticking to the Plan

In some parts of Ireland country roads are either completely empty or full of sheep. Get stuck behind a flock, yes that's the collective term apparently, of sheep and you just have to hope they're being driven to a field nearby. A strip of road that would have taken you 5 minutes to travel could now take you 30. However deviating from the GPS and taking an alternate route could save you time. It probably won't take you the 5 minutes the original route was, the GPS generally plots the quickest route, but it may not take you the 30 minutes that sticking to the sheep filled road would. Turning onto the road and allowing the GPS to recalculate may well reveal a better option than the original plan.

Projects are full of potential unknowns (known and unknown) and sometimes you hit roadblocks. Integration pieces may be delayed by a third party not meeting their deadline, hardware requisition may well get bogged down in red tape, people may get actually hit by buses. These things can well cause projects to stall.

Sticking to the plan could result in a significant amount of time where people are twiddling their thumbs waiting for work. Sticking to the plan is not always the best option. In order to keep people moving it may be necessary to find alternative solutions be it in the short or long term.

Changing a plan my seem a bit daunting especially when the impact of sticking to the plan is so vague but a team that is allowed to think and change fast can reduce any associated risk, cost and/or time.

The Big Finish

So we've almost made it to the ferry. It's due to leave in 30 minutes and the ETA on the GPS is showing 35 minutes. Lets allows ourselves to get a wee bit fantastical for this last one.

Cue wacky races music.

You've gotta make up time so you push that big red button on the dash marked "In Case of Emergency" and the boosters on the back of your car spring to life. As the G-forces press you into the back of your seat the car careers off the road through barns full of hay, chicken shacks full of eggs, back gardens full of laundry, fences, walls and just about every other trope in 70's cartoons. Just as the ferry is a few meters from the port you and the car make an amazing jump onto the only spare parking space on the ship.

The car covered in sheets, eggs, hay and dirt sags with exhaustion its engine sputters to a halt and a few bits fall of it. You and your travelling companions step out dazed, little birds flutter around your heads. You made it. Now the next leg of your journey can begin.

Most projects have some sort of timed delivery of features. It could be weeks, months or even years (shudder!) but irrespective of the time there will invariably be crunch times. Some worse than others depending on the project, the customer or just your companies approach to project delivery. The one thing about crunch time is that you move into "quick win" territory. Quality and "doing it right" become much less important as time becomes the enemy. "Ship it" and "just get it done" are the teams mantras now. It happens. I'm not saying its should be condoned but even on the best projects sometimes things align in such a way that it happens.

The problem is as quality goes down and "good enough" becomes significantly "less good" technical debt, bugs and other types of friction leak in. So once you push that deliverable over the finish line you need to ensure you have time after the fact to go back and ensure you clean up any technical debt and edge case bugs you may have introduced.

If you don't allow the team to address these issues you begin layering layers of debt on top of each other and one day someone will have to repay it. It may not be you, it may not even be a member of your team, but leaving a mess for someone else to clean up is just not something a decent human being should do. You want to be a decent human right?

Finishing up

I think I've stretched this analogy further than it really should have been stretched but I wanted to be as holistic as possible to the idea of team delivery.

As the old saying goes "Its not the destination, It's the glory of the ride"³. While it may seem, from a single customers point of view, that delivering the software is the be all an end all it's much more nuanced than that. It's about building relationships far more than software because it makes delivery of the end product much easier and increases the chance of delivering exactly what people need. It's about ensuring that your team are supported, allowed to take risks and further themselves both individually and as a team. Happy customers will come back for more, happy teams will stick around. Sure every project may not be all sweetness and light but people know this and it's how we deal with the challenges that make the difference.

^{1: At least for the benefit of this post. They are probably entirely different. Go with it.}

^{2: "Resource Planning" - not even once.}

^{3: Edward Monkton, Zen Dog}

Provisioning Postgres via Docker

Fri May 22 00:00:00 UTC 2015

Previously we looked at provisioning Postgres via Vagrant and Ansible and in the spirit of trying something a bit different lets do the same thing using Docker.

This was new territory for me but this guide will get you far enough for hacking around on a project. If this is an awful way of doing things let me know.

Same rules apply - we want to provision a project level Postgres instance listening on host port 5432 with a new database and user.

boot2docker

If you're not using boot2docker (if you're on a supported Linux platform) you can skip this bit

Docker is primarily a Linux based tool but there are wrappers for Windows and OSX. Since I'm using a Mac I'll also use boot2docker that uses a Linux VM and proxies requests to the VM.

I'm going to assume that at this point we have docker and, if necessary, boot2docker installed. If you're using OSX brew is your friend (brew install boot2docker).

With boot2docker installed we need to start it.

boot2docker init # only need this the first time
boot2docker up

After this has started you should see an information message about setting the necessary environment variables to allow docker and boot2docker to talk correctly. There is a quick way to do all this provided by boot2docker

$(boot2docker shellinit)

Dockerfile

The Dockerfile specifies the rules for building a docker image that we can use to create container instances from. I'll just dump ours here and break it down after.

FROM ubuntu

# add postgresql release repository to apt
RUN apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys B97B0AFCAA1A47F044F244A07FCC7D46ACCC4CF8
RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main" > /etc/apt/sources.list.d/pgdg.list

# install the necesarry packages
RUN apt-get update && \
  apt-get install -y \
    python-software-properties \
    software-properties-common \
    postgresql-9.4

# run the rest as the postgres user
USER postgres

# create the database and user
RUN /etc/init.d/postgresql start && \
    psql --command "CREATE USER docker WITH SUPERUSER PASSWORD 'docker';" && \
    createdb -O docker docker

# ensure host can connect to postgres correctly
RUN echo "host all  all    0.0.0.0/0  md5" >> /etc/postgresql/9.4/main/pg_hba.conf
RUN echo "listen_addresses='*'" >> /etc/postgresql/9.4/main/postgresql.conf

# expose the 5432 port to outside the container
EXPOSE 5432

# set the default command to run when starting the container
CMD ["/usr/lib/postgresql/9.4/bin/postgres", \
      "-D", "/var/lib/postgresql/9.4/main",  \
      "-c", "config_file=/etc/postgresql/9.4/main/postgresql.conf"]

The flow is somewhat similar to the previous post,

We want to use the latest version of Postgres so we add the offical Postgres repo (and pgp-key) to the apt catalogue.
We install all the necessary packages
Then we switch to the newly created postgres user to perform the rest of the operations
In RUN we start the database, create the user docker with password docker and finally create the docker database.
We then update the necessary *.conf files to allow us to connect to the Postgres instance outside of the container.
Next we EXPOSE the 5432 port to the outside world (in a boot2docker world this exposes it from the container to the VM, we still have a bit of work to do to get it exposed to the actual host.
Finally we stipulate how to start the Postgres instance when the container starts.

Running

So now we have specified the rules in the Dockerfile (this is somewhat similar to the Ansible playbook concept, sort of, but not quite) we want to build a versioned image from this,

docker build --tag postgres:9.4 --rm  .

Running this from the same directory as the Dockerfile will build an image called postgres tagged with 9.4. The --rm command tells docker to remove the intermediary images created.

We can see that our image has been created using the docker images command

>  docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
postgres            9.4                 f25b0a52a81b        26 minutes ago      358.8 MB

Next we need to take this image and run it as a container,

docker run -p 5432:5432 --name postgres-instance -d postgres:9.4

Lets have a look at the option here,

-p 5432:5432 publishes the EXPOSEd 5432 port effectively making it accessible outside the container.
--name postgres-instance is the friendly name we want to give the container
-d postgres:9.4 is the image name and tag to create the container from

We can verify the container instance is running using the docker ps command

> docker ps
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS                    NAMES
f54de34e1308        postgres:9.4        "/usr/lib/postgresql   22 minutes ago      Up 22 minutes       0.0.0.0:5432->5432/tcp   postgres-instance

At this point we should be able to access the Postgres instance via the IP of the VM. To ge the IP we can use the ip command of boot2docker

> boot2docker ip
192.168.59.103

boot2docker Port Forwarding

If we are using boot2docker we still wont be able to access the instance via localhost. As already mentioned boot2docker transparently uses a Linux VM (via VirtualBox) to run docker within. By default this VM doesn't forward any ports to the host so we need to tell boot2docker (via VBoxManage) to open that port.

VBoxManage modifyvm "boot2docker-vm" --natpf1 "postgres-port,tcp,127.0.0.1,5432,,5432"

This command tells VirtualBox to forward the 5432 TCP port of the boot2docker VM to the host.

At this point we should be able to access the Postgres instance via localhost on the host as well as the VMs IP address directly.

Provisioning Postgres via Vagrant and Ansible

Wed May 20 00:00:00 UTC 2015

This article covers a simple workflow for setting up a Vagrant instance running Postgres provisioned by Ansible. The first time I did this I found that most articles didn't cover everything and I had to look around to solve a few minor issues.

It's worth noting that I can avoid all this and use Nic Ferriers very useful Postgres Dev Box. This work may be unnecessary but it's still fun - Funecessary

I'm going to make an assumption that you know what both Vagrant and Ansible are and you understand the basic premise of how they work. I'll also assume that you have ansible installed on the host machine (Ansible is purely SSH and doesn't need agents installed on target machines).

Vagrantfile

We start with the Vagrantfile. This is the file that tells Vagrant what VM base box to use (Ubuntu Precise 32), what ports should be forwarded to the host (5432 - the default Postgres port) and what provider should be used to provision the VM once it has been brought up (Ansible).

If you already have a local instance of Postgres running you'll want to shut it down or you'll get port collisions and Vagrant will complain.

Vagrant.configure("2") do |config|
    config.vm.box = "precise32"
    config.vm.box_url = "http://files.vagrantup.com/precise32.box"

    config.vm.network "forwarded_port", guest: 5432, host: 5432

    config.vm.provision "ansible" do |ansible|
        ansible.playbook = "playbook.yml"
    end
end

playbook.yml

Ansible provisioning in Vagrant accepts, at the minimum, a playbook to run and the provider takes care of ensuring that ansible is called correctly. Our Vagrantfile above is configured to point to playbook.yml that sits alongside the Vagrantfile so we need to create that.

- hosts: all
  sudo: yes
  gather_facts: no
  tasks:
    - name: add keyserver to apt
      command: apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys B97B0AFCAA1A47F044F244A07FCC7D46ACCC4CF8
    - name: add custom postgres repo to apt
      lineinfile: dest=/etc/apt/sources.list.d/pgdg.list
                  line="deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main" state=present
                  create=yes

- hosts: all
  sudo: yes
  gather_facts: no
  tasks:
    - name: ensure apt cache is up to date
      apt: update_cache=yes
    - name: ensure packages are installed
      apt: name={{item}}
      with_items:
          - python-software-properties
          - software-properties-common
          - libpq-dev
          - python-psycopg2
          - postgresql-9.4

- hosts: all
  sudo: yes
  sudo_user: postgres
  gather_facts: no

  handlers:
    - name: restart postgresql
      service: name=postgresql state=restarted

  tasks:
    - name: postgresql should listen on all ports
      lineinfile: dest=/etc/postgresql/9.4/main/postgresql.conf
                  regexp="^listen_addresses"
                  line="listen_addresses = '*'" state=present

    - name: postgresql should allow access to host
      copy:
        dest: /etc/postgresql/9.4/main/pg_hba.conf
        content: |
          local   all   postgres   peer
          local   all   all        peer
          host    all   all        0.0.0.0/0   md5
      notify: restart postgresql

- hosts: all
  sudo: yes
  sudo_user: postgres
  gather_facts: no

  vars:
    dbname: vagrant
    dbuser: vagrant
    dbpassword: vagrant

  tasks:
    - name: ensure database is created
      postgresql_db: name={{dbname}}

    - name: ensure user has access to database
      postgresql_user: db={{dbname}} name={{dbuser}} password={{dbpassword}} priv=ALL

It probably best to tease apart what each part of this playbook does. I've broken the playbook out into four distinct sections that perform specific related tasks to make this easier to explain,

Add the Postgres apt repository to allow us to get the latest release of Postgres (apt by default will install 9.1 but I want 9.4
Update the apt cache and install the necessary packages (ultimately postgresql-9.4 but we need a few base packages as well.

By this point Postgres will be installed and a postgres user created but if you try and connect to the instance from outside the VM it still wont work. We have some additional work to do. This is exactly what the next block does.

Update the postgresql.conf file to allow Postgres to listen on all ports and update pg_hba.conf to allow the host machine to connect to Postgres. NB these rules are very lax, for a bit of local development work this is fine but it would be best to tighten these up for more serious work. The service should be restarted after both of these tasks.

So now we have the ability to connect to the database from the host machine we also want to set up a dedicated database and user.

Create a vagrant database and create and associate a vagrant user to this database.

If we save both files and run vagrant up we will end up with a provisioned Vagrant box running Postgres that can be access from the host machine on port 5432.

The Builder Pattern

Fri May 15 00:00:00 UTC 2015

The Builder Pattern

Wed May 13 00:00:00 UTC 2015

This article is adapted from a presentation I gave on the Builder Pattern for Deloitte Digital on Friday the 15th May 2015.

Lets talk a bit about the Builder Pattern. But why the builder pattern in particular? Well being a bit of a technical butterfly jumping around from language to language, technology to technology, there are a surprisingly small amount of patterns that port well. So I tend to fall into patterns when writing code more than actually thinking "oh this could be solved using the XYZ pattern to decouple the doo-hickey from the weehoo". The builder pattern, however, I've always found to be very portable across technology and language domains. It may not exist in most domains to solve the original problem it was created for but it is really rather versatile and can be used to create cleaner code. It is also a very applicable when constructing representations of your data such as HTML, XML, JSON etc. and this is an area you're unlikely to avoid in any project regardless of technology or language.

Originally the builder pattern was created to find a solution to the problem of the telescoping constructor anti-pattern but it has grown beyond this use over time. That said defining the original scope of the problem will help us understand the builder pattern better.

Here we have the definition of Reading class. It represents some sort of discrete reading from a sensor or something similar. It has a number of immutable properties and a single constructor for creating an instance of a Reading.

Now lets imagine all of these Reading properties are optional and can be defined with default values. So we can begin by filling in constructors to support easy construction of this class with defaults.

With this constructor we call the previous "full" constructor passing a default value in. This is the first link in a chain of constructor calls.

Our next constructor calls the previous one passing a default date which results in that constructor calling its previous one passing in it's default value as well. We now have our second link.

And so this continues on for a while as we gradually create a longer chain of constructor calls

Successively adding another constructor to the chain of constructors.

You'll notice that only a single default value is passed to the next constructor so we aren't duplicating construction logic and default values.

If we zoom out and look at an outline of our class constructors we can get a better picture of what has happened.

If you've ever owned or just seen a collapsible telescope you can see where the telescoping constructor anti-pattern gets its name.

So at this point we have a few minor problems

We have taken a simple class and filled it full of boilerplate.
We have coupled all the constructors into a chain which makes the class rather brittle which can make refactoring slow and prone to error.
We've scattered our default values across the class. While each value only has a single point of definition the declarations are decentralised.

If you think about it for a while you'll likely see a few simple ways to remove these issues without having to resort to nothing more than a bit of tweaking and refactoring of the current solution.

But lets consider some other scenarios.

So what happens when we want to support constructing a Reading and we only want to set the time to a non-default value?

Well that was fairly easy - but we have created yet another constructor. Not the end of the world but we have diverted slightly from our original constructor chain creating another branch of behaviours to test. We've also duplicated default value logic across more than one constructor. Again not the end of the world and we could pull the values out into constants but we've certainly made everything less clean.

What happens then if we want to introduce a constructor that only sets the sourceId property? Now we have a problem,

We already have a constructor that takes a single int for id. Working around this may require exposing the knowledge of default values outside the class and have the constructing object know to pass in -1 to the 2 arg constructor but then we have externalised coupling and what's the point of having a nice encapsulated OO design if you end up doing something like this?

Those among you who use sensible languages that have features such as default parameter values and copy constructors will find this isn't as much of a problem but even in those languages its possible to tie yourself up in knots.

As I originally mentioned the builder pattern is remarkably versatile and isn't just used to solve this one anti-pattern. There are a number of benefits to be gained by applying the builder pattern to code.

In many situations you achieve more concise, expressive code which can aid readability and maintenance of the code
You can use it to compose complex objects without having to fully understand how the object is constructed internally
You can optimise object creation. The builder pattern can be both immediate and lazy (performing the building steps only when needed) which allows you to defer potentially costly steps to the last minute, or pipeline and batch certain steps where it gives a benefit.

To demonstrate this lets take a look at a method for building a customer order.

Here we have a fairly procedural view of creating an order, adding some line items and marking it for priority shipping.

There is a lot going on and it's up to this method to know exactly how all these bits and pieces fit together. Some things could go wrong,

We may forget to call .addLine() for an item creating a bug
Product.find could be an expensive operation and we are doing it a few times.

An alternative, employing the builder pattern, could look similar to this,

There is much less code here, its subjectively easier to read and understand and there are a few more benefits.

You don't need to understand exactly how line items are added and you can't accidentally miss out an addLine call.
We don't call Product.find ourselves, instead the builder knows to call it and it may be possible for the builder to selectively batch these calls to reduce the overhead.

One more very common use of the builder pattern is through the construction of representations of our data and communication across domains.

To demonstrate this overly wordy phrase lets take a look at a few examples.

Here is one example of a Java DSL that is used to construct a SQL query. The fluent syntax gives you a consistent language to express your intent and, once complete, the builder can take all the knowledge and convert it into a SQL dialect. A huge benefit to this approach is that it removes the need to manually guard against SQL injection attacks.

Another example, this time form Scala, uses the builder pattern, albeit rather minimally to allow us to build a JSON representation of a piece of data.

Although it's not wise to end on a low note it is worth mentioning that as with all things in this world the builder pattern has its own share of issues.

The 2 biggest causes for concern are,

Using the builder pattern to create a DSL such as the SQL example above can result in rather complex builder code. Typically builders have their fair share of boilerplate to support their API, however, this is an acceptable trade off because we've can reduce boilerplate and complexity in our business logic but edge cases can a desire to make the perfect DSL can cause complexity creeep. Ensuring you're not generating unnecessary complexity is important when creating a builder.
As we've witnessed in our very first example (Lomboks builder didn’t set the default values for our optional fields) it is sometimes possible to use a builder and yet create "incomplete" objects missing values that need to be set. If your builder is part of a library that is used across projects then this could cause frustration. Documentation and correct handling of invalid states is essential.

And that is the builder pattern in a nutshell. It is very likely you'll have already come across uses of the builder pattern in the wild. It is a very simple and flexible pattern but should, as with all things, be treated with respect.

Less Java with Lombok

Fri May 08 00:00:00 UTC 2015

So Java, you've heard of it right? It's the language thats used to convert XML into stack traces. You know the one... it's really popular and has a massive ecosystem of powerful open source tools and techs but everyone says it's the worst. Sadly making fun of Java lacks any real challenge because, if I can be blunt, Java makes it easy to do so (see I just did it without thinking).

There are a lot of things that make Java an easy target but in my frequent exposure to Java there are 3 things that really wind me up.

Type erasure & Non-reified generics lead to a whole host of frustrating problems when you're trying to build nice clean composable solutions,
Following on from #1 the type system in general is a real pain. It's neither powerful enough to use it in any real meaningful way nor is it loose enough to allow dynamic typing. It's basically an annoying landlord who happens to live with you, in your bed, and likes farting and being the little spoon.
Finally the sheer verbosity of the whole thing. When 90% of your code is boilerplate classes and methods required to work around limitations in the language you really need to work to make the simplest of code expressive and maintainable.

But I've already said it is popular and interestingly its popularity has been growing. Probably in part due to the power of the JVM and the huge powerful sea of tooling and tech that exist for it. That said I have a theory that it also acts as a sort of natural constraint that in some ways help prevent language astronauts code golf functionality into an unreadable one line functional expression that looks like a computer novice tried to quit Vim. This is all a personal theory formed after reviewing a fairly substantial amount of Scala code, YMMV.

Any way... Without switching to another language there is very little we can do about #1 & #2 so at the very least addressing #3 could make your time in Java more tolerable. Java 8 added a few features that can improve the expressiveness of your code including Functional Interfaces and Default Methods but by and large we are still producing a large volume of boilerplate. These are the challenges with having a great backward compatibility story.

I put on my robe and wizard hat

Fear ye not you beautifully desperate people there are things you can do. A word of caution first - there is a certain amount of "magic" coming up. Yes magic is a bit dodgy because it requires wizards to control but there are levels of magic and a lot of it is just quick slight of hand. Once you're shown the reveal you can't unsee it. Admittedly I was a skeptic of what I'm about to write about but after spending a bit of time with it I approve of considered and careful wielding of this magic.

Lombok

So lets remove the boilerplate from Java without having to rewrite our solutions in a new language. To do this we are going to use Lombok. Now the website is very light on explanation but Lombok is an annotation based code generation library that focuses on removing the noise and boilerplate from your code without the need for actual authentigenuine magic. So while it may feel like magic being released from the shackles of Javas boilerplate oppression it is just a bunch of common patterns rolled into annotations. Oh and yes, if you've read any of my older posts, annotations are a pet hate of mine so you'll understand I'm not saying any of this lightly.

So whats the elevator pitch for Lombok (after I've spent over 500 words meandering over nonsense)? OK lets go.

Assume we have a Character (in the RPG sense not the textual sense) class that has just 2 properties - a name and a level. Now the name is set once and never changes and the level fluctuates as this character kills dragons, marries trolls and harvests magic mushrooms (usually it goes up but you can never really tell with magic mushrooms).

Got it? Good.

This Character class is, and you'll forgive me for using such a stupid term - a DTO. It's a bag of state that gets passed around your application. This happens A LOT in Java applications because of a crappy type system. If you know Java well enough you'll understand that a big-bag-o-state/DTO needs decent toString, equals and hashCode implementations because that thing is going to get logged and compared like a demon (surprisingly demons get compared and logged more than you'd think - I guess hell really is full of Java developers). The implementation of such a class would look like this.... brace yourself.... POJO is coming...

public class Character {
  private final String name;
  private int level;

  public Character(String name) {
    this.name = name;
  }

  public String getName() {
    return this.name;
  }

  public int getLevel() {
    return this.level;
  }

  public void setLevel(int level) {
    this.level = level;
  }

  public boolean equals(Object o) {
    if (o == this) return true;
    if (!(o instanceof Character)) return false;
    final Character other = (Character) o;
    if (!other.canEqual((Object) this)) return false;
    final Object this$name = this.name;
    final Object other$name = other.name;
    if (this$name == null ? other$name != null : !this$name.equals(other$name)) return false;
    if (this.level != other.level) return false;
    return true;
  }

  public int hashCode() {
    final int PRIME = 59;
    int result = 1;
    final Object $name = this.name;
    result = result * PRIME + ($name == null ? 0 : $name.hashCode());
    result = result * PRIME + this.level;
    return result;
  }

  protected boolean canEqual(Object other) {
    return other instanceof Character;
  }

  public String toString() {
      return "se.yobriefca.tinkering.Character(name=" + this.name + ", level=" + this.level + ")";
  }
}

Now as a developer I care about maybe 2 or 3 of those 50 lines of code. With Lombok you can do this...

import lombok.Data;

@Data
public class Character {
  private final String name;
  private int level;
}

Oh look, squint a bit and we have a Scala case class or Kotlin data class but we are still writing the Javas. Neat right ¹? One annotation, completely constrained to that class and we've eliminated a lot of boilerplate and sprinkled in some "best practise" (the hashCode and toString stuff is often sadly neglected by many developers).

But wait! Maybe we want to support creating a Character with some sort of initial level as well as a name. OK, here goes.

@Data
@AllArgsConstructor
@RequiredArgsConstructor
public class Character {
  private final String name;
  private int level;
}

What I've done is, via annotations, add both an AllArgsConstructor and a RequiredArgsConstructor. The names are fairly self explanatory but we can now instantiate our instances like so,

Character c1 = new Character("James", 50);

Or,

Character c2 = new Character("James");

A HA! But what if we have a more complicated object with a plethora of required and optional fields? Well Lombok also gives you an annotation that generates a fluent builder interface for your object.

So this,

@Builder
public class Character {
  private final String name;
  private int level;
}

Gives you this,

Character c3 = Character.builder()
  .level(50)
  .name("James")
  .build();

Of course you can mix and match these annotations to a certain extent to achieve exactly what you need to build your lovely little property bag. As an unsurprising bonus this is all done in a consistent and predictable way that works very well with most 3rd party libraries such as serialisation libraries or data access libraries.

But wait - there's more - it's not all about them POJOs/DTOs/State Bags lets look at some of the interesting things you can also do with Lombok.

Type Inference

Declaring variables in Java can be an exercise in repeating yourself multiple times. For example,

final ValueOutputterProvider<String> valueOutputterProvider = new ValueOutputterProvider<String>();

It's kind of obvious what this is because it says so three times which is not only wasteful and slow to type but its yet more unnecessary noise that reduces readability. Lombok allows us to shorten this declaration making it look more like a Scala declaration

val valueOutputterProvider = new ValueOutputterProvider<String>();

Not exactly earth shattering but it certainly reduces the noise we have to tune out to see the important code.

@Log (inc. @Slf4j @Log4j etc.)

You know that line you write every time you create a new class? You know the one that allows you to log things?

public class WorkerThing {
  private static final Logger log = LoggerFactory.getLogger(WorkerThing);

  public void doWork(){
    log.info("Doing work");
  }
}

That one on the second line. We can make it go away with Lombok,

@Slf4j
public class WorkerThing {
  public void doWork(){
    log.info("Doing work");
  }
}

We've traded a single long line of declaration for a single annotation. Clean. There is an annotation for each of the common logging libraries.

Cleanup

You know those pesky try/finally blocks that we wrap around resources to ensure we don't leak connections or keep streams open unnecessarily?

DBConnection connection = new DBConnection();
try {
  connection.execute("DROP TABLE users");
} finally {
  connection.close();
}

If we are using Java 1.7 onwards we could use the try-with-resources statement to make it cleaner,

try(DBConnection connection = new DBConnection()) {
  connection.execute("DROP TABLE users");
}

But Lombok gives us another option - the @Cleanup annotation,

@Cleanup DBConnection connection = new DBConnection();
connection.execute("DROP TABLE users");

The @Cleanup annotation generates the necessary boilerplate to ensure we are dutifully closing any resource we create even when something goes wrong.

Experimental

Lombok has a bunch of features that are still tagged as experimental. These features either have known issues or have yet to prove themselves thoroughly in the field of Java-ing. I'll demonstrate one of them now - @ExtensionMethod.

I really like C# extension methods, Kotlins extension methods, Scalas extension methods (implicit def's) and so on. @ExtensionMethod gives you this feature for Java.

First we create a static suite of helper methods,

public class StringExtensions {
  public static String makeMusical(String in) {
    return "¸¸?·¯·?¸¸?·¯·?¸¸"+ in + "¸¸?·¯·?¸¸?·¯·?¸¸";
  }
}

Then we annotate that class we want to utilise the new extension methods in,

@ExtensionMethod({StringExtensions.class})
public class App {
  public static void main(String[] args) {
  }
}

Finally we call the static methods within this class as if they were instance methods of the type that the method accepts.

  public static void main(String[] args) {
    System.out.println("EXTEEEEEENSION METHODSSSSSS".makeMusical());
  }

Under the hood the call simply gets translated to

StringExtensions.makeMusical("EXTEEEEEENSION METHODSSSSSS")

Which gives us the rather pretty output of,

¸¸?·¯·?¸¸?·¯·?¸¸EXTEEEEEENSION METHODSSSSSS¸¸?·¯·?¸¸?·¯·?¸¸

In fact @ExtensionMethod can take any class with static methods and make them callable as instance methods such as java.util.Arrays.

OK extension methods are just a nice bit of syntactic sugar but they can be really nice when creating a consistent approach to code style.

Tooling Support

In short "yes there is". Eclipse and IntelliJ both have Lombok plugins with powerful refactoring support (turn Lombok'd classes into their equivalent pure Java class and vice versa). Some of the features, especially the experimental ones, have some little issues that can make editors complain, even though the project will compile and run fine, but it's no worse than some of the bizarre syntax issues you can get with Scala for example.

Summing up

I've thrown a lot of code around and didn't cover all Lombok has to offer. I focused on the features that I think are handy to use and I would certainly recommend you apply you're use of Lombok carefully. As with all powerful things don't over do it just because you can. A little goes a long way.

Finally I recommend you check out the official Project Lombok site, read the caveats on the various features that catch your eye and just give it a go yourself.

^{1: There are some other things generated here such as the bizarre java.beans.ConstructorProperties annotation. I pray I never need to know what that does.}

Not All Frameworks

Wed May 06 00:00:00 UTC 2015

It's interesting, to me at least, that many people in the software industry have very black and white opinions. Maybe its the fact we wrangle binary (albeit at a very very high level these days) on a daily basis that gives rise to such binary views, or perhaps its reflective of the relative young age of the industry (and the mentality this fosters). Who knows.

Sadly it does reflect, often negatively, in the solutions we produce. I've worked in legacy apps that are littered with the snake oil covered spent cases of silver bullets and read my fair share of "Why we are using <%=arbitrary_new_shiny_thing%>" posts followed, in time, by "Actually were going back to an older reliable tech because <%=reasons%>". Then of course there is the wasteland of abandoned open source projects written in yesterdays techs while the sole creator is belligerently re-writing everything in todays tech because it's going to change the shape of IT.

I know I've been guilty of many of these things (and probably will be again) but in my older age I've started to care less about arguments that are, more or less, irrelevant. For example any argument that ignores the most important aspect of technology application, that of context, immediately has me thinking [CITATION NEEDED]. Without context its very easy to validate your own argument like so...

Find the worst examples of the things you disagree with. Hearsay and rumour is also acceptable.
Counter this information with the best examples of the things you agree with. Drop in words like performant, webscale and intuitive and you're bound to walk away feeling self-satisfied.

Your "opponent" can do exactly the same thing and achieve the same results. It's like magic.

This seems to be the current state of affairs when it comes to talking about frameworks vs libraries. In fact I think it always has been. The frameworks vs libraries debate is one of those topics that is always very black and white, usually focused on specific technologies. The general theme is that frameworks are bad and libraries are good and therefore anything that could be categorised as a framework is quickly blacklisted in favour of a few libraries no matter how mature the framework is or how broken the library is.

"But frameworks don't compose" - You know what bucko, neither does most of your crappy code. "Libraries give me flexibility" - yes they do but can you say with confidence that you know what you are doing? Libraries aren't as ready to deal with cross cutting concerns in your solution and lets be honest do we really want to be sitting around designing login/registration/forgot password workflow every time we create an app? The yaks have been built and the bike sheds shaved.

But the biggest problem I see with peoples "frameworks are evil" mindset is that the definition is far from perfect. The definition I like the most is very coarse grained and open to interpretation which isn't ideal but it feels acceptable,

With a library your code is responsible for calling the library whereas with a framework your code is executed by a framework within the context of the framework.

Good enough.

But as I alluded to - there are libraries that are clearly libraries and there are frameworks that are clearly frameworks - but the other 90% live in this amazing fuzzy area I typically call practicality. With a dash of context many of these technologies can be viewed in a different light. In fact we've even invented this term, one that over the course of my time in IT I've started to really dislike, micro-frameworks. We've been told that frameworks are pure evil and will ruin your software project (when in fact its usually people) but, you know, as a starting point for a project or service libraries don't quite have enough opinion so instead we'll create a term that makes us feel better about the whole thing - a micro-framework. It's like a library but with opinion. One of the greatest examples of this abuse of verbage is Spring Boot - it's a fine framework that helps hide a lot of friction and complexity that comes with the Spring web stack. It has been tagged with the micro-framework label more times than I cared to count. But thats the thing, almost nothing has been taken away from it's core functionality as a Spring MVC framework - it has simply chosen to hide it behind carefully constructed opinion and default behaviour but it is still hiding the behemoth that is Spring.

So Spring Boot takes a framework and makes it a micro-framework (it doesn't really of course but there you go). What about the other end of the spectrum? Lets say you just started a nice shiny greenfield project and people are all up in your face about microservices so you start out building some of these fancy microservices. In time you'll start to notice common usage patterns, common behaviour or technical requirements. Being the clever spod that you are you'll start extracting these patterns and behaviours out so they can be reused across your services. Eventually you'll refine a lot of it so that parts of the lifecycle of your services are completely managed by these patterns. Now you've only gone and built a framework haven't you? Clearly that was bad design, back to the drawing board old bean eh? No. Look at Dropwizard as another example. A nicely curated set of well documented and tested libraries thrown together with a bit of glue code (again opinion) - all in all you end up at the same place you did with Spring Boot (again, not quite but superficially speaking you have). One is a set of curated libraries and the other is a single mass of curated classes - it's all in the packaging.

A framework gives you consistency and predictably - these are often good things. Decisions you shouldn't waste time making are made for you, stuff you aren't aware of yet are being dealt with while you remain blissfully unaware, security is a big one, frameworks often deal with matters of security quite well. Libraries, not so much. Its all in the wrist, it's how you use it that makes the difference. Depending on any 3rd party code too much creates nasty coupling and you should avoid this.

A wonderful example of avoiding 3rd party coupling is shown in The Destroy All Software episode #10: Fast Tests with and Without Rails by Gary Bernhardt. In the episode he shows how you can get faster test cycles by isolating model code within Rails. This isolation means the model itself has no reliance on Rails (making the startup times for tests much faster) and yet can still utilise the powerful features of Rails when necessary.

In any project of sufficient scale you'll likely end up creating something akin to a framework anyway. It could be loving handcrafted from libraries over time or you could get immediate productivity by adopting a suitable ready made framework. The choice is yours because you have the context available to make an informed decision. So look - saying frameworks are bad and libraries good is missing the point. It's missing context. It's also missing the fact that many good frameworks are good because they give flexibility similar to libraries but without having to make common decisions over and over. That sort of quality isn't easy to develop so don't go judging all frameworks as if they were evil. Attempting to pigeon hole any particular technology is also a wasted effort because its so easy to subvert the standard definitions.

In short - use what works, don't tightly couple yourself with any third party code (library or framework), don't be quick to judge, listen to the biased nonsensewaffle that most people speak with skepticism and think. Please, please, above all else, think.

Quick Markdown Preview in Vim

Thu Apr 30 00:00:00 UTC 2015

A while back a rather excellent person (@tosborn) posted an article on previewing files in your terminal. The concept is simple. Using Pandoc we can convert the markdown file to HTML and pipe the result into Lynx a terminal based browser. While this doesn't exactly give you an accurate representation of what the output will look like in a graphical browser it gives you an overall quick view of the overall layout and structure. Handy when you are building up a README for a library or tool.

What I'm going to show is using the same technique to render markdown files but do it without leaving the editor (assuming your editor is Vim). I'm going to assume you have both Pandoc and Lynx installed (the article linked above will explain how to do this for Mac).

I'm a Vim user (when I can) and when I'm working with markdown files it's usually in the context of making quick edits and verifying them. To achieve this I've lifted the original process outlined in the link above and mapped it to a key mapping in Vim,

:map <Leader>x :w<cr>:!pandoc % \| lynx -stdin<cr>:redraw!<cr>

Lets break this down a tiny bit,

:map <Leader>x - map the following to keystroke,x (, is my leader key in vim)
:w<cr> - save the current file
:! - start executing a shell command
pandoc % - put the current file % through pandoc
\| lynx -stdin<cr> - pipe the pandoc output into lynx and issue command
:redraw!<cr> - redraw the vim window (this prevents the "Press Enter to continue prompt")

So hitting ,x gives me a quick look at how the markdown file would look without leaving the terminal.

Don't Call It A Comeback

Wed Apr 29 00:00:00 UTC 2015

Taking a break is only a break if you get back into it. Otherwise thats just retiring.

Activity on this site has declined recently. Initially it was intentional (in 2014 I decided to reduce my speaking engagements and focus on contracting), take a break, clear the head etc. but I got sidetracked and kept meaning to get back to a bit of writing and speaking. This proved harder than usual. Travelling every week kind of knocks the wind out of you, or me at least.

In an effort to shine a light on the lack of activity I created a contribution chart to show the drop off and, interestingly enough, I've been in a bit of a dip since before I actively decided to take a break. Or perhaps I'm just finding my rhythm. Or perhaps consistency isn't my strong point, you know like MongoDB.

Anyway the plan is to get back at it, talk more, write more and rant more. I've missed it.

For the interested the contribution chart was thrown together using D3 a hacked version of the calendar view example and CSV data generated every time this site generates.

Notes on Wake

Mon Feb 16 00:00:00 UTC 2015

I've spent the morning tinkering with Wake because I had time and it popped up on Twitter at the right time.

It's an interesting language that aims to be

a fast, expressive, type safe language that gives you testability from the ground up.

What follows is a write up of my thoughts from the short time I've spent with Wake.

Baked in Dependency Injection

I have plenty of experience in Java and have seen my fair share of codebases that make extensive use of dependency injection. The verbosity of Java has created many a monster. Providers, Factories, FactoryProviders and ProviderFactoryProviders fill project hierarchies all in an attempt to make the language bend to our will.

The main problem, aside from the obvious verbosity of Java, is its lack of syntax to support dependency injection. Vast technologies have been built around this simple concept alone. However without the language level support you are forced to go for either wizardry (see @Inject) or verbosity (through additional annotations) - neither of which are desirable.

One of Wakes fundamental features is the use of provisions which is essentially a means to bind dependencies to particular behaviour. Classes define exactly what they need and provide in order to get the job done.

import Printer;

every Main is:

  needs Printer out;

  main() {
    out.printLine("Hello");
  }

In the example above we can see that Main explicitly needs Printer. When the application is executed an instance of Printer will be injected into our Main. In this case Wake knows how to inject Printer as it is from the standard library. We can also provide our own dependencies including any rules required to construct them.

import Printer;

every Main is:

  needs
    Printer;

  provides
    Person <- Person(Printer, ?Text),
    Printer;

  main() {
    (Person("James") from this).sayHi("Internet");
  }

every Person is:

  needs
    Printer out,
    Text    me;

  sayHi(Text otherPerson) {
    out.printLine("Hello " + otherPerson + " I'm " + me + "!");
  }

There is a bit more code here so let me break down some of the important lines.

We have a Person class. This class needs both a Printer and some Text that represents the persons name.
The Person class has a void method that takes a single Text argument and uses the injected Printer to print a greeting.
The Main class, as before, needs a Printer
This time the Main class also provides a few things. It provides a Printer which I will expand on shortly and it also provides a Person type. More so it specifies a way to construct a Person instance without having to specify a Printer each time - Person(Printer, ?Text). In essence this is saying when Main is asked to provide and instance of Person always use this particular Printer instance and take a Text argument
Our Main class is called via the main() entry point and at this point creates a new Person instance using the Person provider defined in Main (hence the use of this - (Person("James") from this))
We call sayHi on this provided instance.

It may seem initially weird for an object to define its own needs and provides but this need not be the case. It is possible to create discrete provider classes that encapsulate construction behaviour which can be injected into our target class. So instead of from this we could say, for instance, from PersonProvider.

This syntax allows us to create defined contracts for our classes without a lot of extra verbosity that could mask intent. It also gives us certain guarantees around type safety and compile time checking the our object graphs are correctly defined. Finally, as an added bonus, this syntax gives us a rather clean way of wiring up stubs and mocks during testing by simply swapping out our providers with stubbed providers.

Testing

On the subject of testing I was a little surprised to read "gives you testability from the ground up" and to not see any examples of how to write tests. Of course the language is still quite young and things are changing. After a bit of poking around I discovered tests are annotation based,

import Asserts;

@TestClass
every MyFirstTest is:

  @Test
  itIsWhatItIs(Asserts) {
    var Text = "test";
    Asserts.that(Text)Equals("test");
  }

Running make will run all the discovered tests in the tests/ folder. Fairly straightforward.

Other Nuances

There are few things in Wake that exist in an attempt to make code more concise, more readable and potentially drive design down a particular route.

Naming Variables

Most variables do not need to be given a name. Our Person class above could be written in a more concise manner,

every Person is:

  needs Printer, Text;

  sayHi($Text) {
    Printer.printLine("Hello " + $Text + " I'm " + Text + "!");
  }

We have one instance of Printer so we don't need to give it a variable name and can refer to it directly by its type.
The same applies for Text however as we have 2 Text instances we can make use of the $ symbol to avoid conflicts.

The argument here is that the type of a variable is often the most meaningful name. This is easily a very contentious point which makes sense in some situations but not necessarily in others. Thankfully its possible to give variables names which I did in the first instance in order to avoid introducing too many new concepts in one go. This approach can take you down the "thinking in types" path where you make the types more rich rather than just place holders for rich data.

Method Names

Like in Swift and Objective-C, Wake methods names don't need to follow the () structure. Arguments can in fact be spread throughout the name. While it may seem odd at first it can make complex methods a bit clearer.

sayHelloTo(Text)As($Text) {
  Printer.printLine("Hello " + Text + " I'm " + $Text + "!");
}

main() {
  sayHelloTo("Emma")As("James");
}

Wrapping Up

I've spent more time putting this article together than I have spent with the language but all in all there are some compelling features in Wake.

The contract based approach of needs and provides makes the structure of your application very clear and aids testing

The ability to reduce the volume of code by eliminating unnecessary cruft (variable names for example), where it fits, goes a step further to allowing us to create more maintainable code.

As of right now the documentation has some holes in it (for example around testing) and the language is still very young but I think its a language worth keeping an eye on.

I've only scratched the surface of the language and its future plans so I recommend, if this is the first time you're reading about Wake, you check out the site for more information - http://www.wakelang.com/

The Twelve Factor App

Tue Jan 20 00:00:00 UTC 2015

Yesql - SQL in SQL in Clojure

Tue Nov 25 00:00:00 UTC 2014

Fair warning. This was initially a transcript for a yesql screencast I intended to do but time is dragging on and I really can't see myself getting to that any time soon.
Why does this matter?
Well I wouldn't be surprised if this reads pretty much like the yesql README.md with some minor embellishments here and there (and perhaps some opinion). But I'm putting it out there just, well, just because. Yesql is lovely and consider this a tribute of sorts.

Yesql is a Clojure library for accessing SQL capable databases using SQL. Unlike its contemporaries yesql, as the name alludes to, embraces SQL. The argument here is why create a DSL, no matter how beautifully crafted, when we already have a perfect DSL already in the form of SQL.

To go a bit further lets look at the rationale in yesql's own README.

You could write your data access like this, as embedded SQL strings,

(query "SELECT * FROM users WHERE country_code = ?" "GB")

But this doesn't scale very well,

commands can get scattered about the code base,
its hard to see syntax errors, and,
longer statements can be difficult to read without any sort of syntax highlighting.

The next logical step taken to solve this problem is by adding an additional layer of abstraction. Language specific DSLs are created to dynamically build SQL.

(select :*
        (from :users)
        (where (= :country_code "GB")))

The benefit these DSLs have is that they remove the need to know the specific dialect of SQL you're using. As each database has a slightly different SQL dialect you simply specify an adapter, such as PostgresAdapater) and the translated SQL will magically be Postgres compatible. Moving to a different database engine is a matter of specifying a new adapter with no code changes.

Sounds good right? Well not quite - as with almost all abstractions they start to break down when you tread off the happy path.

Every database system will likely have additional, non-core features, that you'll probably want to exploit, in fact these features are usually factored into selection criteria when you were deciding what to use. Postgres array and JSON data types are a good example of this. These are features that your clever DSL abstraction can't support without extra wrangling because they address the generality of problems. Suddenly you're back to writing raw sql for some of the queries and we are back where we started.

Yesql tackles these problems by putting SQL in SQL files and allows you to reference these blobs of SQL as clojure functions. So you'll lose the ability to execute the same code across different databases, which if my experience is anything to go by isn't the worst thing that can happen, but you'll gain better editor support, correct syntax highlighting and one less abstraction to ruin your mood.

Seeing it in action is the best way to demonstrate its simple but powerful api. We'll start by cloning a ready rolled project template

git clone -b 00-starting-point git@github.com:yobriefcasts/007-yesql.git

This is nothing more than a project generated by lein new with 2 dependencies added,

One is the Yesql library dependency itself, currently at version 0.4.0, and
The other is the h2 database driver which we will use as our database in these examples

Finally we have a file based h2 database instance which is pre-populated with a people table containing a few rows of dummy data.

1	James	Hughes	james@yobriefca.se
2	Emma	Hughes	emma@yobriefca.se
3	Ollie	Hughes	ollie@yobriefca.se
4	Nate	Hughes	nate@yobriefca.se

The First Query

So lets get this data out of the database.

As is common across most SQL based libraries in Clojure Yesql derives its connections from a db-spec. This is the same db-spec map you'd pass Clojure's JDBC library or even korma. I'll provide a link to the relevant documentation if you want more information.

The db-spec required to connect to the current h2 database is already defined in the core.clj file.

(def db-spec {:classname    "org.h2.Driver"
              :subprotocol  "h2"
              :subname      "./resources/demo.db"
              :user         "sa"
              :password     "" })

So now if we want to create a SQL file get-all-people.sql under the src/sql directory we can write a simple select * statement from the people table.

select * from people

Its worth noting that the sql folder name isn't necessary I simply used it for neatness. By default when calling Yesql - paths will be relative to the src directory so you can organise it however you want.

Now we need to call this from our code. You'll notice in core.clj that I have already included a single require for defquery from the yesql.core namespace.

defquery is one of the 2 macros that gives you an entry point into Yesql, we will get to the other one shortly. defquery takes a name and a path to the query file. This allows defquery to parse the SQL file, convert it into a callable function and define a var based on the name you've given it.

We can define our get-all-people query like so,

(defquery get-all-people "sql/get-all-people.sql")

Although I've kept them the same the name of the file and the query name don't need to match. If we evaluate this line now we get back our def'd function. Subsequent evaluations, for example if you've changed the contents of the SQL, will result in the def being redefined.

If we call our new function as-is we'll get an error.

(get-all-people)

This is because we also need to supply the db-spec that was already defined for us. If we call the function correctly we should see our results.

(get-all-people db-spec)

Lets give our output a bit more structure. We can import clojure.pprints pprint function

(ns yesql-tinkering.core
  (:require [yesql.core :refer [defquery]]
            [clojure.pprint :refer [pprint]]))

And run our query again.

(pprint (get-all-people db-spec))

Passing data to queries

Executing static SQL isn't going to get you very far, if it does and you are only writing read-only systems with ready rolled queries - can I please come work with you? At the very least your queries often require WHERE clauses that need to take values based on some user input from outside the database.

To demonstrate how we do this in Yesql lets add a new query that filters by surname. We can start by creating a new SQL file src/sql/get-people-by-surname.sql and write our query. As with most frameworks that make use of parameterised queries we put token in place of the value we want to pass in, in the case of Yesql this can be a keyword that will associate a descriptive name against the argument..

select * from people where last_name = :last_name

Next we can use the defquery macro to "realise" this query.

(defquery get-people-by-surname "sql/get-people-by-surname.sql")

We can call this as we previous did except we need to pass it an argument for last_name.

(pprint (get-people-by-surname db-spec "Hughes"))

Documentation

Lets diverge a little from our path. Imagine a more complex SQL statement that accepts more than one argument, lets say it has a very domain specific name like isolate_aggregated_upstream_resources - how would you 1. find out what this statement does and 2. how you would call it correctly? Well most obviously you could simply look at the SQL and work it out but that's not really an effective way to go about things.

In out last example SQL statement we used a parameter name of last_name, well this wasn't just for fun, if well look at the documentation for our query we can see that the parameter name is used as an argument name to our function

(clojure.repl/doc get-people-by-surname)

This is handy but we can go one step further and supply doc compatible documentation in our SQL in the form of comments. By prepending a comment on top of our SQL we can dive into supplying additional commentary around what this SQL is actually doing.

-- Find all people for a given surname

If we evaluate our defquery again and look at the function signature we should see our comment available as a docstring.

(clojure.repl/doc get-people-by-surname)

yesql-tinkering.core/get-people-by-surname
([db last_name])
  Find all people for a given surname

Grouping queries

The one file per query thing is all well and good until it isn't. You could rather quickly end up with a raft of folders with 3 or 4 queries each which is a bit of a pants way of arranging things.

Yesql provides a way to group similar queries in a single file, well they don't have to be similar but lets assume we have some logical grouping here.

So lets add a few accessor methods for getting people out of our database. If we create a SQL file src/sql/get-persons.sql and add 2 queries, one for getting a user by ID and another for getting a user by email address. When you use defquery you specify the name of the function in the declaration and the entire file is used as a basis for the query however when we are defining multiple queries at once we need both logically separate the queries in the file and associate a name with each one. We do this by adding a structured comment above each query starting with name: and followed by the name you want to refer to the query as in your code.

-- name: get-person-by-id
select * from people where id = :id

-- name: get-person-by-email
select * from people where email_address = :email_address

To define these queries in our code we need to import the other macro in yesqls arsenal defqueries. We can then use this to import our queries,

(defqueries "sql/get-persons.sql")

You'll notice that defqueries returns a vector of the defined functions and we can see our two function handles. We can call each of these in the usual way.

(get-person-by-id db-spec 1)
(get-person-by-email db-spec "james@yobriefca.se")

Non-Queries

We've still only dealt with querying our database but most systems usually need to manipulate the contained data by either writing new rows or updating and deleting existing rows.

Rather than add a new set of macros - yesql uses naming conventions to signal to the macros that the operation is a non-query operation.

For example if we create a SQL file containing a statement for inserting a new user that expects first_name, last_name and email_address, the database will auto generate our ID value.

insert into people(first_name, last_name, email_address)
values(:first_name, :last_name, :email_address)

We can use defquery again to create our function except this time we prepend the name with an exclamation mark. This is the marker to tell yesql that this is a statement rather than a query.

(defquery insert-person! "sql/insert-person.sql")

The function expects the usual db-spec object as well as our arguments

(clojure.repl/doc insert-person!)

And we can call it in the normal way.

(insert-person! db-spec "Bill" "Thompson" "bill@thompson.com")

This style can be used with any non-query statement such as create, update, delete, alter and all those other statements your database engine supports.

Statements will return the number of rows affected by default however in the case of insert statements against tables that have auto-generated ids you may want the id of the newly inserted row returned instead. You can achieve this in yesql with the less than exclamation (<!) instead. If we we redefine our insert-person! function to use this we can see that it will return a different set of information

(defquery insert-person<! "sql/insert-person.sql")

(insert-person<! db-spec "Emma" "Thompson" "emma@thompson.com")

What is actually returned depends on the database engine, h2 returns a map containing the id but some engines return the full row.

Wrap Up

Thats just about covers the main features of yesql which as you can see is a neat little solution to bridging the gap between the database and your code. It avoids all the normal pain of wrangling with other peoples conventions and leaky abstractions and allows you to use SQL directly which is a much better medium to express queries against relational data stores.

Preflight Checks

Thu Oct 09 00:00:00 UTC 2014

What I'm about to describe is in no way a new thing, you'll find evidence of this sort of pattern everywhere. I've noticed some people do it instinctively and others simply fall into the trap that this sort of technique/pattern tries to avoid. Therefore I think its worth a bit of a mention.

An Example in a Different Context

Lets explore the problem in a slightly different context - dynamic vs static languages. Imagine the given scenario in JavaScript

function longest(a, b){
  return a.length < b.length ? b : a;
}

longest("aa", "b");  // "aa"
longest("a", "bc");  // "bc"
longest(1, "bc");    // 1

We've defined a function called longest that returns the longest string argument between the two passed to it. We also defined a couple of sample uses of the function,

"aa" is longer than "b" so "aa" gets returned
"bc" is longer than "a" so "bc" gets returned
Here be dragons!

In the last example our return value isn't what we expected. At best we'd expect "bc", with a length of 2, to be longer than 1, whose string value would be "1" and therefore have a length of 1. But that isn't the case. Numbers don't have a length property and therefore the result of 1.length is undefined. Comparing undefined to 2 is going to yield false, hence our unexpected return value.

Lets look at this problem in another language, one that is statically typed - TypeScript,

function longest(a: string, b: string) {
  return a.length < b.length ? b : a;
}

longest("aa", "b");
longest("a", "bb");
longest(1, "bb");

Trying to compile this sample will fail on the last line because a number isn't a string. This happens long before you get to run your code.

The Problem, Solution and Result

So what's the problem? Well in the dynamic example we won't know about the issue until such times as the application gets into a certain state and the manifest problem may happen elsewhere far far down the call chain causing great confusion and frustration.

Solutions to this problem are available.

In static languages we actually have to bend the type system to encounter such problems because we have type checking and compilation up front providing a certain level of application correctness.
In dynamic languages we avoid these issues by writing tests that (should) ensure correctness of our application before we run them.

Ultimately being able to know something is wrong as early as possible is essential to making releases pain free and predictable. In turn, being able to release with confidence allows you to release more often and tighten those vital product feedback loops.

The Problem at a Higher Level

Now lets move away from code and compiler level concerns and think about your application as a deployable thing. Your app will no doubt have dependencies on external services, datastores, configuration or other applications within the same host. All of these things will cause your app to go into some form of failure state should they be unavailable (lets leave aside the fact that services can go down during the course of a running application, thats a different matter entirely). The issue may not be immediately obvious either - a misconfigured application that you call may only present errors for a given set of requirements and finding out whats wrong in a production or other similarly closed off environment can be a a test of any ones patience.

Being able to detect, as early as possible; like we do with a compiler or test suite, if our applications environment is in a valid state can save a lot of pain further down the line.

Some of you may be ready to tell me that your favourite provisioning tool (Puppet, Chef, Ansible, Salt etc.) gives you certain guarantees already and that is true to a certain extent but they are often not a sure fire guarantee.

Nothing is perfect and sometimes even the most tested tools behave differently
You may not have it set up correctly to deal with certain unique states of deployment boxes
You may have ultra-secure environments that you can't even run these tools on without a lot of pain and so manual provisioning is necessary.

Additionally if you are following any of the points in the Twelve Factor App Methodology you'll notice you are relying more on the hosting environment and this can amplify the problem.

The Preflight Check

The pre-flight check pattern aims to address these problems by ensuring invalid states in application dependencies fail deployment in a way that lights a flare above the problem so it can be fixed.

A few things led me to develop this habit/pattern

I was deploying smaller services to Heroku that promotes pushing config down to the OS
I was a standard Play! stack which often has less-than-helpful error messages when things go wrong during startup
I was provisioning things using vagrant and puppet/chef/ansible which, during the early stages, I often forgot to update as the applications needs evolved.

Ultimately I got bored of opaque stacktraces and bizarre behaviour in odd edge cases and so attempted to ensure my hosting environment was in good shape before we "took off" so to speak.

In short a pre-flight check is code that runs when your application starts, asserts the state of the environment or external services/applications. If something fails the code will halt startup with a meaningful message that can be used to quickly diagnose the problem.

In Play! this took the form of a hook in Global.scala

object Global extends GlobalSettings {
  override def onStart(app: Application) {
    performPreflightChecks()
  }

  def performPreflightChecks() {
    Option(System.getenv("DATABASE_URL")).getOrElse {
        Logger.error("[PREFLIGHT_CHECK] DATABASE_URL is not set")
        throw new Exception("[PREFLIGHT_CHECK] DATABASE_URL is not set")
    }
  }
}

This is an exceptionally primitive example (that I've written without the aid of a compiler so it may be incorrect) - in fact as the pattern developed I eventually extracted the preflight check pattern into it's own plugin that performed a number of things,

Ran a number of checks (classes that extended Runnable) within a configurable namespace.
Logged a report of checks that passed and failed (regardless of if everything passed or not)
Exposed a method that can be called in tests - where deployment environments also supported running test directly.

Some of the checks I've used have been,

Ensuring ImageMagick is correctly installed with the correct codecs (difficult to get right even if you use a provisioning tool I've found)
Ensuring System environment variables are set correctly
Database connectivity
OpenCV JNI bindings available

Of course, this is in no way limited to Play! or Heroku, I've used this across a number of technologies (Clojure, Ruby, Dropwizard etc.) and environments (locally, vagrant, AWS, Rackspace, Heroku etc.) and the time tackling silly oversights has been greatly reduced.

Many technologies already have the capacity to perform preflight checks in a way that integrates with your ops story (Dropwizards healthchecks are a perfect example) but preflight checks would mandate running these checks as the application starts as well.

In short, regardless of technology or platform, it is essential that deployments validate application and hosting environment state before going live. The preflight check pattern aims to address this issue by ensuring you map out all you dependencies of your intended host irrespective of where that may be.

Operation Sprint Finish

Sun Oct 05 00:00:00 UTC 2014

A few months back I closed off a busy period. Between normal 9-5 working and running some workshops and talks, I didn't have much free to devote to side projects and my writing. As soon as things died down I was ready to dive into all those ideas that I had logged and pushed to the back of my mind. Thats about as far as I got. Thinking. Little to no action, and even when I did muster the motivation to actually start something I either got distracted with a different; fresher idea or over thought the problem so hard that I either reduced it to something that already exists or over complicated it that it lost the appeal. A lot of these are signs of burnout so I decided to take some time to clear my head. I've done little to no blogging, accidentally ran a Clojure workshop and generally thought about how to approach the problem of tackling my ideas rather than the ideas themselves.

I am now into my third week of what I've playfully dubbed Operation Sprint Finish and the reason why I'm talking about this now is that this week is my writing week - a week purely devoted to writing about all those topics I have listed in Wunderlist (though not necessarily publishing everything this week). Based on the last 2 weeks I've genuinely made decent progress on two of my personally important ideas than I did prior to having structure.

So what is Operation Sprint Finish? Well I first started by listing as many of the ideas I've had as possible (I ended up with about 20 or so that I really wanted to tackle). Next I selected 5 of these and designated them as immediate and the rest as sometime in the future. Well technically I selected 6 but that also includes the ever present writing, screencasting and presenting work (which all fall under the same high level category - or if you're into agile buzzwords - epic). These main ideas are the first batch of ideas I want to take as far as I logically can before I determine them either redundant or release them in some form. From these six epics I generated a list of things that need done and could be done reasonably within a one week period. Most of these were marked as Idea XYZ Sprint #1, Idea ABC Sprint #1 etc. with some overarching objectives and goals. Then using REALSCIENCE I picked one of the items and moved that into play for the next week/sprint.

For one week I focus squarely on that task. If new things come up they go on the backlog, new ideas go on the sometime in the future list (or swapped with one of the immediate epics as there can be only 6). At the end of the week I leave a comment (I'm using Trello to orchestrate all this) on the epic so that I can pick up any future tasks quickly and pick my next item. In fact I seem to find picking the tasks for the next 2 weeks at once a rather nice incentive as I know what to look forward to.

From the first two weeks I've managed to build

A slightly more opinionated Luminus project structure that gives me a great starting point for the app I was working on as well as future web apps using a Clojure stack.
A semicomplete prototype for an Android app (using Kotlin which was a pleasure) and a lot of design ideas for the app which was very frustrating but also rather fulfiling.

Both of these weeks have resulted in results that are far more complete than I've managed in recent times.

And now this week I can start putting some articles on my site that I've been itching to talk about for a while. Perhaps even record this screencast I've been threating to do.

What I've done is allow myself to not be distracted, not talk myself out of an idea because the idea on the the other side is greener and worst case, should I hit a hard part (like how to approach the user experience of the mobile app I'm working on) I don't just give up and instead think it through and potentially crack it (which I think I did).

Indeed I expect many people to be saying "So what? You're just applying the agile-y stuff to getting side projects done" and thats true but I still think there is value in writing about it. Besides, at the very least, this article still counts towards my current sprint deliverable which I guess is something, right?

Functional Programming with Clojure

Tue Sep 23 00:00:00 UTC 2014

Tyrannosaurus Rx

Wed Sep 03 00:00:00 UTC 2014

Simple stubbing with lein-cooper and lein-oneoff

Tue Aug 26 00:00:00 UTC 2014

Work on lein-cooper has been pretty quiet. It's not because it has been abandoned but because it does enough for my needs. As and when it starts getting issues or pull requests coming in I can start adapting it to fit the wider needs of others.

Recently I came across a lovely leiningen plugin called lein-oneoff. oneoff allows you to specify dependencies for a clojure script at the top of a clojure script (a bit like what Grape does in the Groovy world I believe). This means you don't have to spin up all the project infrastructure to run a simple script. So instead of lein new server and editing the core.clj file to mock up a simple "Hello World" sample you could do something like this,

(defdeps
  [[org.clojure/clojure "1.6.0"]
   [ring/ring-jetty-adapter "1.3.0"]
   [ring "1.3.1"]])

(ns server
  (:require [ring.adapter.jetty :refer [run-jetty]]))

(defn handler [request]
  {:status 200
   :headers {"Content-Type" "text/html"}
   :body "Hello World"})

(run-jetty handler { :port 3000 })

If this were saved as server.clj we could use lein-oneoff (declared in the global ~/.lein/profiles.clj) to run it like so,

> lein oneoff server.clj

Notice the defdeps form at the top of the file, this will be used to grab dependencies and set up the appropriate classpath for running the script.

Pretty neat.

I also noticed that this would be a perfect accompaniment for a typical usage of lein-cooper - stubbing out external services. If you're building a system that integrates with other services (either internal or external) you introduce a bit of friction when it comes to developing and testing (at the integration level at least) your system. Either you have to build and run dependent services or rely on external services that may or may not be available. A simple solution is to develop against simple stubs for these services. This is what coop-off (the new power couple portmanteau I've given to the lein-cooper/lein-oneoff combination) can make easier.

Just create a stubs folder in your project, create standalone service stubs (like the example below that also demonstrates the use of oneoffs *command-line-args* for accessing arguments passed with the command),

(defdeps
  [[org.clojure/clojure "1.6.0"]
   [ring/ring-jetty-adapter "1.3.0"]
   [org.clojure/data.json "0.2.5"]
   [ring "1.3.1"]])

(ns server
  (:require [ring.adapter.jetty :refer [run-jetty]]
            [clojure.data.json :as json]))

(defn handler [request]
  { :status 200
    :headers {"Content-Type" "application/json"}
    :body (json/write-str { :success true :data [1 2 3 4] }) })

(def port
  (or (Integer. (first *command-line-args*)) 3000))

(defn start-server
  [port]
  (run-jetty handler { :port port }))

(start-server port)

And you can integrate the running of this and other stubs alongside using lein-cooper. Just create a Procfile for lein-cooper with commands to run everything,

web: lein ring server
data: lein oneoff stubs/data.clj 3000
auth: lein oneoff stubs/auth.clj 3002

Finally run the whole thing via lein cooper.

The standalone stubs will be executed alongside your own service without the overhead of having to manage separate projects or integrate test code into your own project.

Invasion of Privacy

Mon Aug 25 00:00:00 UTC 2014

There is a pattern I see more and more often in Java projects that use Spring as an IoC container thingymajig. I doubt it's limited to Spring and probably common among other IoC containers but I don't really get a lot of exposure to others because,

When it's my own project I generally don't need one, and,
When it's a project I'm involved in most people tend toward Spring (bar that one time Guice was preferred).

So don't consider this a rant against Spring and more around a common pattern that seems to be creeping in to many codebases.

But yes, the problem, check this out.

@Component
public class MyBestService {

	@Inject
	private ADependency dependency;

	public void doAThing() {
		dependency.doStuff();
		// more things
	}
}

I see this in high ranking Google search results and documentation as an accepted standard approach. Problem is this introduces two very terrible problems.

It hides dependency knowledge
It poisons your domain with technical design considerations
It bleeds across the codebase.

Hiding Knowledge

This is easily the worst problem IMHO. Without peering into this the code for this class there is simply no way to know what this class depends on. Even then there is little way to know if the dependency is required or optional.

The minute you remove explicitness from code you've started down a dangerous path. It becomes impossible to reason about a system without knowing the entire system. The friction this causes grows with time.

Poisoning your Domain

If you are injecting private fields into your classes you'll probably need some sort of mechanism to do this. Enter annotations. Commonly you'll see @Inject from the javax.inject namespace being used. While @Inject is more generic than framework specific annotations it is still a technical consideration it isn't something related to your domain. Your classes should be pure in the sense they should be simple and reflect the domain rather than how you built it. The mechanism for IoC is completely irrelevant to your solution.

Worse, now you've made injection a special case, your tests need built in a special way - @InjectMocks springs to mind (no pun intended).

Bleeding Across the Codebase

When you use private field injection you see @Inject everywhere. Jumping to what is injected is a matter of understanding the dependency chain which is spread across multiple files. There is no centralised pool of knowledge about how your system is wired together and this makes it harder to reason about.

Remediation

First and foremost build your system without a DI framework as a first class citizen. Of course use IoC to prevent heavy coupling and potential God classes but do it au naturale.

Secondly use constructor injection. Somehow the people leaning on frameworks forgot about constructor injection. They also forgot about setter injection but I've often found optional dependencies are also a bit of a design smell so I'm not really recommending them. Of course the frameworks themselves already support constructor injection so it won't be a big leap to change your evil ways.

So this,

@Component
public class MyBestService {

	@Inject
	private ADependency dependency;

	public void doAThing() {
		dependency.doStuff();
	}
}

Becomes,

public class MyBestService {

	private ADependency dependency;

	public MyBestService(ADependency dependency) {
		this.dependency = dependency;
	}
	
	public void doAThing() {
		dependency.doStuff();
	}
}

Constructor injection is beautifully explicit. You can't construct a new instance of a class without explicitly declaring its dependencies - its right there for all the world to see. You could argue that this makes passing many dependencies very noisy. You'd be right. Bad design should be noisy and passing 30 dependencies into a constructor is bad design. This is something @Inject can easily hide as you add more and more dependencies to a class as there is no requirement for it to be centralised.

This has a knock on effect on your tests too - it makes them normal. You are no longer bound by whatever framework you use to inject mocks and you can explicitly pass stubs without having to understand any particular magic. Or - if you're still so inclined you can still use your framework of choice.

Finally use a centralised way to bootstrap your application. Spring has AppConfig, Guice has Module and framework-less stacks use new in the right place, you do remember new right? This allows you to push the logic for constructing your dependency graph into a central place that can and should be readable making it much easier to understand the design of your system and refactor easier.

So there you go - use containers if it makes your life easier but try and avoid making your code either magic or noisy.

React

Wed Jul 30 00:00:00 UTC 2014

Nomadic Tendencies

Thu Jul 24 00:00:00 UTC 2014

"The future came and went in the mildly discouraging way that futures do."

Neil Gaiman, Good Omens: The Nice and Accurate Prophecies of Agnes Nutter, Witch

If there is one thing that shows the immaturity of the software industry (relative to other industries) it's the ever present need to jump to the next big technology. Tales of "leaving X for Y" dot the path from one trend to the next. "I'm so much more productive in Y over X" is the battle cry of the newly indoctrinated acolytes of Y. Vast swathes of half finished, abandoned open source projects written in X collect dust in their respective repositories sitting alongside new shiny half-finished, soon to be abandoned open source projects written in Y. And the industry marches on.

And yet most software that we use is still awful. It still exhibits the same problems, it still breaks in the most unpredictable ways and it leaves people as angry and frustrated as ever.

But I guess that's to be expected. If we regularly reinvent everything in "the next big thing" when do we ever find time to refine anything? The industry marches on, but the industry never moves. Like a perpetual early beta phase our systems are built up just enough before we tear them down and rewrite them in something new. Even enterprises, who tend to move much slower and are therefore buffered from smaller fluctuations in trend, suffer from the same problems at a different scale.

Ultimately this behavior puts the focus on the technology and how it can make our systems better. Our users are once more second class citizens because what we want is somehow more important (and who needs users anyway when people are throwing millions at pointless smoke and mirror apps).

But the problem is not the technology.

The problem is that we are in love with our tools when we should be in love with our users. We need to talk to our users more and less to our peers. Spend less time on Hacker News debating stuff you only partially understand and try and get a better insight into how you can use your current, very pedestrian, technology stack to genuinely improve the lives of your users.

A Case for Luminus

Mon Jul 21 00:00:00 UTC 2014

First and foremost - frameworks. I view frameworks suspiciously and prefer to build my systems out of a core set of libraries and allow it to evolve and breathe a little without the artificial constraints of a framework. There is a time and place for frameworks, but that time and place never seems to be here and now were my work is concerned. This may be one of the many reasons I find Clojure so enjoyable.

Luminus is a "Clojure web framework". It says so right on the the tagline of the Luminus homepage. This single line has meant I have hesitated diving into Luminus since I starting breaking the back of my Clojure dabbling. I wanted to avoid it until I felt it useful or necessary to invest my time in it, a framework, after-all, needs to be understood as a whole unlike its modular library counterparts.

So my first foray into building web apps in Clojure meant building them out from scratch using a foundation of Ring, Compojure, Hiccup - the usual suspects. Data access meant looking at a few possibilities , assessing each one and making a decision. Various authentication libraries needed thorough inspections. This, as you can expect, leads to a tremendous amount of yak-shaving and time spent satisfying curiosity.

So I had a look at Luminus. I wanted to see what decisions a framework had made and possibly steal some of the better ideas. I used Luminus on my next project. I didn't steal the good ideas for my own work, I didn't need to - the image I have in my head when someone mentions framework is simply not what Luminus is.

In its most basic form a framework is a collection of 3rd party and bespoke libraries tied together with some code that abstracts the library specifics into a more cohesive whole. Luminus doesn't really have this. There is no bespoke modules or libraries, there is no abstraction. Clojars has no luminus library. It does have a Leiningen template that gives you a decent starting point but even that isn't even a heavily opinionated starting point and can be customised to suit your own needs.

So what is Luminus? IMHO Luminus is what you'd end up with if you wanted to personally build a curated list of recommended libraries to build web applications in Clojure. It's what would happen if you bothered to document that process and offer alternatives. In fact while the curated set of libraries makes for a productive development process with reduced decision anxiety it's the documentation that launches it out of the park. Rather than "Do it like this", the documentation offers recommendations and guides for using alternative libraries such as migrations, data access, HTML templating and authentication. Luminus gives you enough to get started and generously gives you a foot up to the next level.

At no point in time does Luminus hide anything from you, it wears its lineage as a badge of honour and won't get offended if you decide it's wrong in places.

I can respect that.

Working with Play apps in the Console

Fri Jul 11 00:00:00 UTC 2014

Having spent a fair amount of time working with Clojure in the last 6 months I've become rather accustomed to the REPL driven development approach. If you're unfamiliar with this approach I recommend reading/viewing Jay Fields post on the topic. Long story short RDD involves writing a small bits of code and executing them immediately. This gives you immediate feedback without having to write tests or start an entire system each time. You can of course make your REPL based experimentation more concrete by saving your work off as a set of tests once your code has settled down a bit.

Like Clojure, and unlike Java, Scala supports the concept of REPL driven development and I've been using it more often since my exposure to its charms in Clojure. Of course Scala syntax is a lot meatier (and I do mean a lot) than Clojure syntax so there is a bit more friction but every now and again its faster when you just want to poke around a concept or idea.

A common example when working with Play is querying the application database with Anorm. The one problem here is that to use Anorm and the Play provided extensions you need to have a running Play instance that it can extract configuration values from.

For example if we tried to get all the users from a database we could write something like this via the console (sbt console or activator console in the root of our project)

scala> import play.api.db._
scala> import anorm._
scala> DB.withConnection { implicit connection => SQL"SELECT * FROM users".map(mapUser).list() }

But this would result in an error.

java.lang.RuntimeException: There is no started application
  at scala.sys.package$.error(package.scala:27)
  at play.api.Play$$anonfun$current$1.apply(Play.scala:71)
  at play.api.Play$$anonfun$current$1.apply(Play.scala:71)
  at scala.Option.getOrElse(Option.scala:120)
  at play.api.Play$.current(Play.scala:71)
  ... 43 elided

Bummer. Thankfully Play provides a quick way to stand up an running instance which can be used in the console

scala> new play.core.StaticApplication(new java.io.File("."))
[info] play - database [default] connected at jdbc:h2:file:./data/db
[info] play - Application started (Prod)
res18: play.core.StaticApplication = play.core.StaticApplication@22a730f2

Now if we try our query again

scala> DB.withConnection { implicit connection => SQL"SELECT * FROM users".map(mapUser).list() }
res19: Seq[scala.collection.immutable.Map[String,String]] = List(Map(username -> kouphax, password -> JHGW$FSWF$KJJK$3231))

To be super clean we can even stop the running instance

scala> play.api.Play.stop()

StaticApplication isn't perfect. For instance it will start your application in production mode but for my needs this has been acceptable. If you do want to provide greater control over how your application is started then look no further than the StaticApplication source.

class StaticApplication(applicationPath: File) extends ApplicationProvider {

  val application = new DefaultApplication(applicationPath, this.getClass.getClassLoader, None, Mode.Prod)

  Play.start(application)

  def get = Success(application)
  def path = applicationPath
}

Using this knowledge you could create your own application instance using DefaultApplication and ask Play to start it using Play.start(application).

Play for (Java|Scala)

Wed Jul 09 00:00:00 UTC 2014

Estimates as Deadlines and the End of Everything

Thu Jun 19 00:00:00 UTC 2014

"If you guess how many USB sticks are in the jar you win an iPad" they said with a smile that had been seared into their face from years of standing at conference popup stands selling the same message.

I glance over at the sacred prize. For randomly picking a number out of thin air I could easily walk away with a brand spanking new...

"Wait", I say noticing something is odd about the prize. "Is that, an original iPad?".

"Well its not a knock off it that's what you mean"

"No no I mean is it an original version, an iPad one?"

"Oh, yep it is"

"And that's what you win?"

"Of course"

"Why are you giving away an ancient iPad?"

"Well no one has ever actually won so we haven't felt the need to replace it"

"Why has no one ever won?"

"Well no one has ever guessed the exact amount of 1GB USB sticks in this jar"

"So you need to be exact, not just be 'the closest'?"

"Of course, now listen buddy, are you just going to question me about the prize or are you going to listen while I talk to you about our latest release of Super Pro Manager 4.0 - Agile Enterprise Edition."

I walk off, its not even worth the effort. I'd either need a the historical data of guesses or a chance to take each stick out and count them individually. Even then the prize at the end of it is completely out of line with what I'd expected in the first place - a new iPad.

The Cold Hard Truth

The story above sounds incredibly silly yet we as an industry still do this on a daily basis.

I'm still hearing, to this day, stories of people treating estimates as targets. I'm hearing them from the enterprise companies that will never change and I'm hearing them from the companies that grace our stages at conferences and talk about how they are "enabling agile teams to deliver value by running lean with a shared responsibility".

The reality is many teams are forced into this fixed template of Scrum with its overhead of extra ceremony and expected to deliver more without giving up the practices of old. It's crunch time every sprint for the team. Its unsustainable. We see the senior members "rise up" and become "cross team" conduits for best practice and status updates. They have successfully extracted themselves from the daily grind and frustration of the sprint.

It may sound like just another one of those developer frustrations that managers tell us "same as it ever was, back in my day..." ad nauseam - but it's not. When we combine it with the faux-agile, process over people, approach most organisations actually have then we have a serious problem. Seriouser, in fact. The teams that were already shipping software in a healthy manner continue to do that, the ones who changed over the last few years in response to careful agile marketering and brand engagement (through certification and top dollar coaching) have just heaped more pain onto their process.

The iterative and incremental application of big design upfront methodology practices will completely and utterly dehumanise a team. People begin to tell themselves that the flat team, the shared vision, one goal stuff was nonsense anyway, it pays the bills for the houses and flats they never really get to see because of the "Unending Project Death March" being exacted one overladen, overprocess-ised, overtime driven sprint at a time. The team don't get their 3 months head space of building solutions against slipping timelines followed by a few weeks of intense crunching, no, they get a constant slightly less intense crunching all the time, non-stop, forever and ever and ever and ever and ever.

Don't believe me? Trust me this is so widespread, so pandemic, that one musical collective even wrote a song about it - Godspeed You! Black Emperor - The Dead Flag Blues. Of course they had to give it a fictional political lean to avoid getting into trouble at work but at the risk of getting them found out allow me to break their brutally honest piece down a bit.

The car is on fire, and there's no driver at the wheel
And the sewers are all muddied with a thousand lonely suicides
And a dark wind blows

This is every IT project. Ever.

The government is corrupt
And we're on so many drugs
With the radio on and the curtains drawn

The "government", yeah right, more like a cheeky nod to managment and sales. The drugs are Scrum and Agile and we are listening to the things these people say without looking outside our walls following our estimates and hoping it will all be different this time.

We're trapped in the belly of this horrible machine
And the machine is bleeding to death

Public and private money spent on failing systems across every industry and we are all just cogs.

The sun has fallen down
And the billboards are all leering
And the flags are all dead at the top of their poles

The flags are the principles we once felt we stood for - making a difference, changing the world, all dead or dying.

It went like this:
The buildings tumbled in on themselves
Mothers clutching babies
Picked through the rubble
And pulled out their hair

Buildings clearly represents the promises and contracts based on estimates. The fallout of these being wrong are clear.

The skyline was beautiful on fire
All twisted metal stretching upwards
Everything washed in a thin orange haze

At some point you start to accept that this is as good as it gets. You're immune to the mess, in love with the decimation you and your contemporaries have wrought. But its all still broken.

I said, "Kiss me, you're beautiful -
These are truly the last days"
You grabbed my hand
And we fell into it
Like a daydream
Or a fever

You give up, you go with it, you accept that broken is the only thing this can ever be. You allow yourself to believe this, you forget everything you ever cared about.

We woke up one morning and fell a little further down
For sure it's the valley of death

You adopt a new technology, a new platform, you let it consume you, you raise your rates.

I open up my wallet
And it's full of blood

Consultancy.

The End of Everything

It may have seemed like a small thing to enforce estimates as targets but you've just ruined the world a bit more. Your teams write crappy broken software because they hate what they do. They hate what we do because they are forced to work against arbitrary timescales like battery chickens - no light, hemmed in pushing out sub-standard cheap bits of code as often as possible. Quantity over quality, no time for quality, no way no how.

This is the end of everything.

Publish and Subscribe with core.async's pub and sub

Wed Jun 04 00:00:00 UTC 2014

Following on from my previous article that covered core.async's merge and mix this article will delve into another suite of complementary high level methods - namely the channel driven publish/subscribe model that core.async can support.

While the pub/sub mechanism provided by core.async is higher level than working directly with chans and mults to achieve the same thing (in fact that is how the pub/sub operations are implemented internally) it should still be considered a set of lower level primitives compared to say a specific event bus notification with topic namespacing, wildcard broadcasting and other such specialised features.

I've attempted to visualise how the various parts of a pub/sub setup would interact. We'll use this as a starting point and expand on how this all falls together through the course of the article.

Setup

Once again we'll use ClojureScript to demonstrate the use of pub/sub. I'll refer you the Setting Up section of my previous article if you want to get a basic ClojureScript setup running with core.async

Publishing

Channels are core.asyncs foundation and this is no different when you move up a level of abstraction to a pub/sub model. Looking at the publishing side first there are 2 main types of actors or components involved,

The publisher channel - Putting stuff on this channel will cause the publishing mechanism to kick in
The publication - calling pub on the publisher (we'll see this in a minute) produces a publication that subscribers can sub/subscribe to.

This may appear a bit more jarring than a typical approach where you just have a monolithic pub/sub object (e.g. a bus) that you publish and subscribe to directly. However I found the idea a bit easier to understand when I thought of the publication as a realised topic or set of topics.

If you think of a simple message bus implementation (in whatever language floats your boat - I've went with Swift, nah just kidding its JavaScript) you would traditionally do something like this,

// grab/create a bus - typically global.
var bus = new Bus();

bus.subscribe("account:created", function(data){
  // do something with the knowledge that
  // an account has been created
});

// at some point we publish to the same topic
bus.publish("acoount:created", {
  success:true,
  username: "petedaleet17"
})

The topic in this example is account:created however in reality its just a string, there is no real confidence that you are achieving what you expect. A spelling error will result in rather unpredictable results and as we have introduced a high degree of decoupling it may well be difficult to track down. The observant will notice there is in fact a spelling error in the code.

In core.async the notion of a publication is a realised entity you perform actions on and pass around. It's the transport mechanism, that is the "concept". This inversion is interesting. 99.9% of the time I'm ultimately interested in publishing or subscribing to a topic, I really don't care about the mechanics of that operation and this topic-first view better supports that thought model.

Now just to confuse things a bit more, a publication also defines a way to derive the topic of a published value so its entirely possible for a publication to handle subscriptions for any number of topics. Another analogy that may help, or just make things worse, is that a publication could be considered a router that defines a set of rules for where to publish data.

To expand on this lets look at some code.

First we need to import a few things into out ClojureScript file (probably core.cljs)

(ns chat.core
  (:require [cljs.core.async :refer [chan <! >! timeout pub sub unsub unsub-all]])
  (:require-macros [cljs.core.async.macros :refer [go]]))

Now lets introduce our publisher and our publication,

; publisher is just a normal channel
(def publisher (chan))

; publication is a thing we subscribe to
(def publication
  (pub publisher #(:topic %)))

You can see the publisher is simply a channel, no fancy annotations or extra functions being applied. It's just a vanilla channel that you can put stuff and take stuff off if you so desired.

The publication ont he other hand is composed by calling pub on publisher passing a function. When you put stuff onto publisher this function is used to derive the topic of the data put onto the channel. It will get passed the put data and is expected to return something, anything that represents the topic.

In this case I am making an assumption that our data will be a map with a :topic entry. Of course if (:topic data) returns nil that will be our topic. Any topics that have no subscribers will be disregarded so nil will typically be an acceptable dead letter office so to speak.

Subscribing

So now we have something we can push data onto - the publisher channel, and something that will accept subscriptions - the publication. Next we need to subscribe and start putting stuff on that our subscribers can consume. We could start putting stuff on the channel now but no one will be around to receive it just yet.

The yin to pubs yang is sub. sub always takes a publication, a topic and a channel (you can also specify how and when the subscribing channel will be closed). As I've already mentioned you'll notice that the subscriber channels don't subscribe directly to the publisher channel. This would allow you to create many publications from a single source channel.

Taking our previous example we could subscribe to a number of topics like this,

; define a bunch of subscribers
(def subscriber-one (chan))
(def subscriber-two (chan))
(def subscriber-three (chan))

; subscribe
(sub publication :account-created subscriber-one)
(sub publication :account-created subscriber-two)
(sub publication :user-logged-in  subscriber-two)
(sub publication :change-page     subscriber-three)

We now have 3 channels

subscriber-one is subscribed to the :account-created topic
subscriber-two is subscribed to both the :account-created topic and the :user-logged-in topic
subscriber-three is subscribed to the :change-page topic

Now when messages are put on to the publisher the publication will inspect the data and determine where to route the message. To make this more visual we can listen to and print the values from these channels by trying to take from them in a go-loop

(defn take-and-print [channel prefix]
  (go-loop []
    (println prefix ": " (<! channel))
    (recur)))

(take-and-print subscriber-one "subscriber-one")
(take-and-print subscriber-two "subscriber-two")
(take-and-print subscriber-three "subscriber-three")

If we start putting messages on the publisher we should see some console output.

(go (>! publisher { :topic :change-page :dest "/#home" }))
; subscriber-three: { :topic :change-page :dest "/#home" }

(go (>! publisher { :topic :account-created :username "billy" }))
; subscriber-one: { :topic :account-created :username "billy" }
; subscriber-two: { :topic :account-created :username "billy" }

(go (>! publisher { :topic :user-logged-in :username "billy" }))
; subscriber-two: { :topic :user-logged-in :username "billy" }

(go (>! publisher { :topic :user-logged-out :username "billy" }))
; No subscribers so nothing to see

Of course subscribing to something is entirely additive. You need some way to take subscriptions away. At this point unsub and unsub-all are what you need.

; unsubscribe subscriber-two from account-created
(unsub publication :account-created subscriber-two)

;test
(go (>! publisher { :topic :account-created :username "billy" }))
; subscriber-one: { :topic :account-created :username "billy" }

; unsubscribe every subscriber from the :account-created topic
(unsub-all publication :account-created)

;test
(go (>! publisher { :topic :account-created :username "billy" }))
; nada

; finally unsubscribe every channel from every topic
(unsub-all publication)

;test
(go (>! publisher { :topic :change-page :dest "/#home" }))
(go (>! publisher { :topic :account-created :username "billy" }))
(go (>! publisher { :topic :user-logged-in :username "billy" }))
(go (>! publisher { :topic :user-logged-out :username "billy" }))
; wonderful silence

Summing Up

So that covers the pub/sub model you get with core.async which gives a slightly higher abstraction from having to deal with the internal logic of managing a mult to perform this operation while still remaining suitably low level to allow you to build upon it for your own needs.

Remember that because everything still boils down to channels, which you have access to, there is huge potential to combine various abstractions to construct simple pipelines for all your needs.

One day I'll finish my channel based Rube Goldberg machine put a message at one end and watch as it gets piped through mults, publications, merged channels, alts and anything else I can think of finally ending up in a console.log...

"Hi James".

Combining & Controlling Channels with core.async's merge and mix

Sun Jun 01 00:00:00 UTC 2014

Most discussions around core.async, be it in Clojure or ClojureScript, tend to focus around the key concepts of the library - specifically chans and the go/go-loop macros. This isn't a bad thing as that is were the power of the library comes from, on the other hand core.async also has a few powerful higher-level features that let you do some very interesting things and they deserve a bit of love as well.

Two such features are merge and mix. Both methods have a similar goal - combining multiple input channels into a single output channel however in practise they are rather different.

At a high level you could draw the 2 operations like so,

In fact the mix diagram is slightly more complicated in reality but we can expand on that as we go.

Setting up

So lets look at some examples of these operations. I'm talking in the context of ClojureScript in this case but same reules and principles apply in plain Clojure.

If you want to try these examples and are rather new to ClojureScript I recently outlined a basic ClojureScript setup which will get you up and running.

Assuming you have an empty ClojureScript project one way or another you need to add a reference to core.async in the :dependencies section of the projects project.clj

  :dependencies [[org.clojure/clojurescript "0.0-2173"]
                 [org.clojure/core.async "0.1.303.0-886421-alpha"]]

Then in your ClojureScript source (probably src/<project_name>/core.cljs) you need to import a few things. I'll assume you have at least a basic understanding of channels and core.async already so we can just import everything we need for the examples.

(ns chat.core
  (:require [cljs.core.async :refer [mix admix toggle merge chan <! >! timeout]])
  (:require-macros [cljs.core.async.macros :refer [go]]))

Now we are about ready for the examples.

`merge`

merge is the simpler of the two features and as the API documentation says combines 1..N source channels and returns a channel which contains all values taken from them. The operation is entirely immutable. That is, once you use merge to create a channel you can't add or remove channels later. When all input channels have closed the merged channel will also close.

This is useful when you have multiple event streams, represented as channels, and you want to process them in the same way and in a centralised manner e.g. when you are wanting to parse multiple simultaneous server requests, web socket events or user interactions from various parts of the user interface.

To keep the example simple we will just create 3 channels that randomly publish their names every now and then,

; declare the channels
(def in-channel-one   (chan))
(def in-channel-two   (chan))
(def in-channel-three (chan))

; define the function for publishing
(defn randomly-constantly
  "Constantly publishes the given value to the given channel in random
   intervals every 0-5 seconds."
  [channel publish-value]
  (go (loop []
        (<! (timeout (* 1000 (rand-int 5))))
        (>! channel publish-value)
        (recur))))

; start putting stuff on the channels
(randomly-constantly in-channel-one   "channel-one")
(randomly-constantly in-channel-two   "channel-two")
(randomly-constantly in-channel-three "channel-three")

So now we have 3 channels that will randomly have their name pushed onto them we now need to do something with them. For the sake of simplicity lets assume all we need to do is log the result. We could write 3 distinct go loops (or suitably abstract it into a reusable function),

(go (loop []
      (println (<! in-channel-one))
      (recur)))

(go (loop []
      (println (<! in-channel-two))
      (recur)))

(go (loop []
      (println (<! in-channel-three))
      (recur)))

But regardles of how much you abstract away the mechanics you are still dealing with the 3 channels as 3 distinct entities when in many cases you should be dealing with a single channel derived from multiple sources. We achieve this with merge

(def merged (merge [in-channel-one
                    in-channel-two
                    in-channel-three]))

merged is now a channel that we can take from and recieve values from all 3 channels. Now we can perform our go loop over the single channel instead,

(go (loop []
      (println (<! merged))
      (recur)))

`mix`

merge is fine when you want to just grab a bunch of channels and treat them as one but sometimes this is not enough. When it comes to channels that produce effects visible to the user there is often a need to better control these messages. Imagine a chat application where each person is represented as a channel, or perhaps a log dashboard where each channel is a service in your system streaming log data.

In such situations, where the volume is high, there may be times you want to focus on a particular set of logs or chat messages, or surpress someone or something that is being particularly chatty. Maybe these messages can be discarded, maybe they are important and need to looked at later. These are the things that merge fails to address. These are the things that mix does address.

The key differences that set mix apart from merge are that,

It introduces an intermediary component - the mixer
It is configurable, you can add and remove input channels
Channels can be muted, paused and solo'ed on demand

So lets take our 3 channels above and apply the abilities of mix to the situation.

First of all we need to create 2 things.

The output channel - unlike merge this isn't created for us
The mixer - we create this via the mix method

; manually declare our output channel
(def output-channel (chan))

; create a mixer linked to the output channel
(def mixer (mix output-channel))

We can also, at this point, set up our go loop for printing the data put onto the output channel

(go (loop []
      (println (<! output-channel))
      (recur)))

Unlike merge we still haven't declared what input channels should be associated with the mixer and ultimately output channel. We can do this using admix,

(admix mixer in-channel-one)
(admix mixer in-channel-two)
(admix mixer in-channel-three)

At this point we should start seeing stuff being logged to the console exactly like we did with merge. This is where mix starts to get interesting.

`toggle`

toggle allows you to control how the mixer responds to each input channel. You pass it a state map of channels and associated mixer properties. With toggle you can do any combination (though many would not make sense) of,

:mute - keep taking from the input channel but discard any taken values
:pause - stop taking from the input channel
:solo - listen only to this (and other :soloed channels). Whether or not the non-soloed channels are muted or paused can be controlled via the solo-mode method.

So lets imagine one of our mixed channels (in-channel-one) it getting a bit chatty. It could swamp our logging output and we might miss something important in another channel. We can use toggle to temporarily mute it,

(toggle mixer {  in-channel-one { :mute true } })

Now our output will only be displaying the other 2 channels. But suppose the data coming in from channel one was actually important, as it stands muting a channel simply discards any takes that happen. If we want to stop taking anything from the channel (and therefore allow it to buffer on the channel) we can pause the channel instead.

(toggle mixer { in-channel-one { :mute  false
                                 :pause true } })

Finally if we want to only concern ourselves with channel one we can solo it

(toggle mixer { in-channel-one { :solo  true } })

You'll notice I didn't set :pause back to false because soloed channels ignore their other properties..

Summing Up

We covered both the merge and mix methods of core.async. Both methods are higher level ways to combine and control multiple input channels into a single unified output channel.

merge offers a simple straigthforward way to combine channels but offers you little control after the fact. mix gives you greater control over the input channels and is exceptionally useful when you need to manage input streams.

Basic ClojureScript Setup

Fri May 30 00:00:00 UTC 2014

ClojureScript is a bit of an awkward one to get into. If Clojure isn't your natural language then you may well find the clojarian setup and the long compilation times a bit of a frustration. Even if you're accustomed to the REPL based development approach that fits so well with Clojure there is still a certain amount of friction to be had with ClojureScript.

Recently a strong desire to play with and better understand core.async in ClojureScript I decided to try and put together a lightweight workflow. In doing so I ended up producing a Clojure library (primrose) and a Leiningen plugin (lein-cooper) to better support this workflow.

It's probably a sure sign that you are doing something wrong when you feel you need to create a bunch of developer focused tools to support a workflow for an already established ecosystem but you gotta do, what you gotta do.

Here goes.

Starting point

ClojureScript projects are still Clojure projects and so there are few bits and pieces that you need to wire up a basic empty project. You've got your project.clj, the lein-cljsbuild plugin for compiling ClojureScript to JavaScript and (at the very least) an index.html for bringing everything together.

Thankfully David Nolen has you covered. Mies is a very basic ClojureScript project template so getting up and running is a matter of,

lein new mies <project name>
cd <project name>

At this point you are good to go. To compile the ClojureScript you can run,

lein cljsbuild once

If you open index.html and view the developer console you should see some predictable output. Alternatively if you run,

lein cljsbuild auto

The plugin will watch your project sources and recompile everything when changes are made. This is significantly faster that manually compiling every time as compilation is incremental. So instead of 10s of seconds for each compile you'll be getting sub-second compilations.

Ajax

When you view HTML files via file:// you won't be able to, at least by default, make ajax calls (due to security concerns such as Same-origin policy).

To fix this you'll want to serve your index.html from a web server. You could go with python -m simplehttpserver as many often do but if you want to keep everything under the same roof (Clojure/Leiningen) then I'd recommend lein-simpleton - a Leiningen plugin for serving static files from the current directory. If you add lein-simpleton to the projects project.clj file,

:plugins [[lein-cljsbuild "1.0.2"]
          [lein-simpleton "1.3.0"]]

You can now run lein simpleton <port> to start serving static files (your index.html). Alternatively if you have actual server side code in the same project you can replace lein-simpleton with lein-ring.

Two for One

So now if you run both

lein cljsbuild auto
lein simpleton <port>

You have a dynamic(ish) environment for working and testing your ClojureScript work.

Problem is I don't like having multiple windows open for long running processes and we now have 2. Ruby has a nice answer for this which plays very well with Heroku - Foreman. Foreman takes a file (Procfile) that contains a list of named command line processes and runs them. These processes are supposed to be long running (like simpleton and cljsbuild auto) and Foreman merges the output of these files into a single stream that makes development and debugging a bit easier.

Again if you want to keep this sort of thing all under one roof I've thrown together a simple plugin for ingesting a Procfile and doing what Foreman does - in Leiningen.

lein-cooper

lein-cooper gives you that Foreman feeling. So lets add this as a plugin.

:plugins [[lein-cljsbuild "1.0.2"]
          [lein-simpleton "1.3.0"]
          [lein-cooper "0.0.1"]]

Now we can create a Procfile at the root of our project to run our two commands.

web: lein simpleton
cljs: lein cljsbuild auto

Now if we run lein cooper we will have single call to start up our dynamic(ish) environment for working with ClojureScript. Once you are done just CTRL-C the terminal and everything will shutdown cleanly.

Summing Up

So we now have a very basic project setup that will allow us to tinker with ClojureScript, make Ajax calls, edit code and cleanly automate away all the stopping and restarting and problematic debugging. It's not quite the REPL driven style of a typical Clojure project but it's minimal enough to try a few smaller scale things.

lein-cooper - Foreman style process runner for Leiningen

Fri May 30 00:00:00 UTC 2014

primrose - Experimental promise/future utility functions

Wed May 28 00:00:00 UTC 2014

Devops: A Case Study

Wed May 28 00:00:00 UTC 2014

The Weird and Wonderful Characters of Clojure

Mon May 19 00:00:00 UTC 2014

This ancient article has been superseded by a much better and up to date article on the Clojure website

A reference collection of characters used in Clojure that are difficult to "google". Descriptions sourced from various blogs, StackOverflow, Learning Clojure and the official Clojure docs - sources attributed where necessary. Type the symbols into the box below to search (or use CTRL-F). Sections not in any particular order but related items are grouped for ease. If I'm wrong or missing anything worthy of inclusion tweet me @kouphax or mail me at james@yobriefca.se.

`#` - Dispatch macro

You'll see this macro character beside another e.g. #( or #". This topic will act as a bit preamble before looking at your specific case.

# is the dispatch macro, a reader macro that tells the Clojure reader (the thing that takes a file of Clojure text and parses it for consumption in the compiler) to go and look at another read table for the definition of the next character - in essence this allows extending default reader behaviour.

Clojure doesn't provide support for creating reader macros but it is possible through a bit of hackery.

If you see # at the end of a symbol then this is used to automatically generate a new symbol. This is useful inside macros to keep macro specifics from leaking into the userspace. A regular let will fail in a macro definition

user=> (defmacro m [] `(let [x 1] x))
#'user/m
user=> (m)
CompilerException java.lang.RuntimeException: Can't let qualified name: user/x, compiling:(NO_SOURCE_PATH:1)

Instead you need to append # to the end of the variable name and let Clojure generate a unique symbol for it.

user=> (defmacro m [] `(let [x# 1] x#))
#'user/m
user=> (m)
1
user=>

If we expand this macro we can see the gensym'd name

user=> (macroexpand '(m))
(let* [x__681__auto__ 1] x__681__auto__)

Another place you'll see the # is in tagged literals. Most commonly you'll see this use in EDN (extensible data notation - a rich data format that can be used in Clojure) and in ClojureScript (#js). Search for #inst, #uuid or #js for some more info.

`#{` - Set macro

See the dispatch (#) macro for additional details.

#{ defines a set (a collection of unique values) specifically a hash-set. The following are equivalent,

user=> #{1 2 3 4}
#{1 2 3 4}
user=> (hash-set 1 2 3 4)
#{1 2 3 4}

Attempting to create a set using this literal form will throw if there are duplicates. Instead the hash-set function should be used on a vector

user=> #{1 2 3 4 1}

IllegalArgumentException Duplicate key: 1  clojure.lang.PersistentHashSet.createWithCheck (PersistentHashSet.java:68)
user=> (set [1 2 3 4 1]) ; convert vector to set, removing duplicates
#{1 2 3 4}

Clojure Documentation: Sets

`#_` - Discard macro

See the dispatch (#) macro for additional details.

#_ tells the reader to ignore the next form completely.

user=> [1 2 3 #_ 4 5]
[1 2 3 5]

The docs suggest that "The form following #_ is completely skipped by the reader. (This is a more complete removal than the comment macro which yields nil).". This can prove useful for debugging situations or for multline comments. I've never used it.

Clojure Documentation - Reader

`#"` - Regular Expression macro

See the dispatch (#) macro for additional details.

#" indicates the start of a regular expression pattern.

user=> (re-matches #"^test$" "test")
"test"

This form is compiled at read time into a java.util.regex.Pattern.

Clojure Documentation: Regex Support

`#(` - Function macro

See the dispatch (#) macro for additional details.

#( begins the short hand syntax for an inline function definition. The following 2 bits of code are similar,

; anonymous function takin a single argument and printing it
(fn [line] (println line)) ;

; anonymous function takin a single argument and printing it - shorthand
#(println %)

The macro expands the shorthand syntax into a function definition whose arity (the number of arguments it takes) is defined by how the % placeholders are declared. See the % character for discussion around arity.

user=> (macroexpand `#(println %))
(fn* [arg] (clojure.core/println arg)) ; argument names shortened for clarity

`#'` - Var macro

#' is the var quote. It is the same a the var method,

user=> (def nine 9)
#'user/nine
user=> nine
9
user=> (var nine)
#'user/nine
user=> #'nine
#'user/nine

When used it will attempt to return the referenced var. This is useful when you want to talk about the reference/declaration instead of the value it represents. See the use of meta in the metadata (^) discussion.

Clojure Official Documentation: Special Forms

`#inst`, `#uuid` & `#js` etc. - tagged literals

Commonly found in EDN and ClojureScript this use of # is called the tagged literal. Look at this example,

user=> (java.util.Date.)
#inst "2014-05-19T19:12:37.925-00:00"

When we create a new date it is represented as a tagged literal, or in this case a tagged string. We can use Clojures read-string to read this back (or use it directly)

user=> (type #inst "2014-05-19T19:12:37.925-00:00")
java.util.Date
(read-string "#inst \"2014-05-19T19:12:37.925-00:00\"")
#inst "2014-05-19T19:12:37.925-00:00"
user=> (type (read-string "#inst \"2014-05-19T19:12:37.925-00:00\""))
java.util.Date

A tagged literal tells the reader how to parse the literal value. Other common uses include #uuid for generating UUIDs and in the ClojureScript world an extremely common use of tagged literals is #js which can be used to convert ClojureScript data structures into JavaScript structures directly.

EDN Tagged Elements

`%` - Argument placeholder

% is not a macro but a placeholder for use in the #( macro. It represents an argument that will be passed into the function when it is expanded.

user=> (macroexpand `#(println %))
(fn* [arg] (clojure.core/println arg)) ; takes a single arg, uses it once

user=> (macroexpand `#(println % %))
(fn* [arg] (clojure.core/println arg arg)) ; takes a single arg, uses it twice

Numbers can be placed directly after the % to indicate the arguments position. Numbers are also used by the #( macro to determine the number of arguments to pass in.

user=> (macroexpand `#(println %1 %2))
(fn* [arg1 arg2] (clojure.core/println arg1 arg2)) ; takes 2 args

user=> (macroexpand `#(println %4))
(fn* [arg1 arg2 arg3 arg4] (clojure.core/println arg4)) ; takes 4 args doesn't use 3

So you don't have to use the arguments but you do need to declare them in the order you'd expect an external caller to pass them in.

% and %1 can be used interchangably,

user=> (macroexpand `#(println % %1)) ; use both % and %1
(fn* [arg1] (clojure.core/println arg1 arg1)) ; still only takes 1 argument

`@` - Deref macro

@ is the deref macro, it is the shorthand equivalent of the deref function so these 2 forms are the same,

user=> (def x (atom 1))
#'user/x
user=> @x
1
user=> (deref x)
1
user=>

@ is used to get the current value of a reference. The above example uses @ to get the current value of an atom but @ can be applied to other things such as futures, delays, promises etc. to force computation and potentially block.

`^` - Metadata

^ is the metadata marker. Metadata is a map of values (with shorthand option) that can be attached to various forms in Clojure. This provides extra information for these forms and can be used for documentation, compilation warnings, typehints and other features.

user=> (def ^{ :debug true } five 5) ; meta map with single boolean value
#'user/five

We can access the metadata by the meta method which should be executed against the declaration itself (rather than the returned value).

user=> (def ^{ :debug true } five 5)
#'user/five
user=> (meta #'five)
{:ns #<Namespace user>, :name five, :column 1, :debug true, :line 1, :file "NO_SOURCE_PATH"}

As we have a single value here we can use a shorthand notation for declaring the metadata ^:name which is useful for flags as the value will be set to true.

user=> (def ^:debug five 5)
#'user/five
user=> (meta #'five)
{:ns #<Namespace user>, :name five, :column 1, :debug true, :line 1, :file "NO_SOURCE_PATH"}

Another use of ^ is for type hints. These are used to tell the compiler what type the value will be and allow it to perform type specific optimisations thus potentially making resultant code a bit faster.

user=> (def ^Integer five 5)
#'user/five
user=> (meta #'five)
{:ns #<Namespace user>, :name five, :column 1, :line 1, :file "NO_SOURCE_PATH", :tag java.lang.Integer}

We can see in that example the :tag property is set.

You can also stack the shorthand notations,

user=> (def ^Integer ^:debug ^:private five 5)
#'user/five
user=> (meta #'five)
{:ns #<Namespace user>, :name five, :column 1, :private true, :debug true, :line 1, :file "NO_SOURCE_PATH", :tag java.lang.Integer}

`'` - Quote macro

Can be used against symbols as part of a dispatch macro (see #'). Also used to quote forms and prevent their evaluation as with the quote function.

user=> (1 3 4) ; fails as it tries to evaluate 1 as a function

ClassCastException java.lang.Long cannot be cast to clojure.lang.IFn  user/eval925 (NO_SOURCE_FILE:1)
user=> '(1 3 4) ; quote
(1 3 4)
user=> (quote (1 2 3)) ; using the longer quote method
(1 2 3)
user=>

Clojure Official Documentation

`;` - Comment

; is a comment. In fact its a comment macro that takes all input from its starting point to the end of the line and ensures the reader ignore it.

user=> (def x "x") ; this is a comment
#'user/x
user=> ; this is a comment too
<returns nothing>

`:` - Keyword

: is the indicator for a Keyword which is an interned string that provides fast comparison and lower memory overhead.

user=> (type :test)
clojure.lang.Keyword

Alternativley you can use keyword to create a keyword from a string

user=> (keyword "test")
:test

A neat thing about keywords is they also implement IFn and can act as functions for extracting values from maps which is very nice.

user=> (def my-map { :one 1 :two 2 })
#'user/my-map
user=> (:one my-map) ; get the value for :one by invoking it as function
1
user=> (:three my-map) ; it can safely access non-keys
nil
user=> (:three my-map 3) ; it can return a default if specified
3

Clojure Official Documentation

`::` - Qualified keyword

:: is used to fully qualify a keyword with the current namespace.

user=> :my-keyword
:my-keyword
user=> ::my-keyword
:user/my-keyword
user=> (= ::my-keyword :my-keyword)
false

I have found this useful when creating macros. If I want to ensure a macro, that calls another method in the macro namespace, correctly expands to call the method I have used ::my-method to refer to the fully qualified name.

What is the :: used for in clojure?

`/` - Namespace separator

Can be the division function / but can also act as a separator in a symbol name to break apart the symbol name and the namespace it resides in my-namepace/utils. This allows symbols to be fully qualified to prevent collisions or spread.

Cloure Official Documentation

`$` - Inner class reference

Used to reference inner classes and interfaces in Java. Seperates the container class name and the inner class name,

(:import (basex.core BaseXClient$EventNotifier)

(defn- build-notifier [notifier-action]
  (reify BaseXClient$EventNotifier
    (notify [this value]
      (notifier-action value))))

EventNotifier is an inner interface of the BaseXClient class which is an imported Java class.

Clojure: Using Java Inner Classes

`-> ->> some-> cond-> as->` etc. - Threading macros

These are threading macros. Almost all of them take an initial value and thread this value through a number of forms. Lets imagine (for reasons unknown) we wanted to take a number, find the square root, cast it to an int, then a string then back to an integer again. We could write it like this,

user=> (Integer. (str (int (Math/sqrt 25))))
5

The threading macro allows us to unravel this deep nesting,

user=> (-> 25 (Math/sqrt) int str Integer.)
5

Or if you prefer multiline and consistent brackettering

(-> 25
    (Math/sqrt)
    (int)
    (str)
    (Integer.))

What the macro does is take the value returned from each expression and push it in as the first argument to the next one.

->> is the same but different. Rather than push the last value in as the first argument it passes it in as the last argument.

The "etc." in the title refers to the fact there are a whole host of threading macros that perform variations on the same theme (cond->, some->, as-> and their ->> equivalents). There is also an entire library, swiss-arrows, dedicated to the threading macros.

`~` - Unquote macro

See ` (syntax quote) for additional information.

~ is unquote. That is within as syntax quoted (`) block ~ will unquote the associated symbol i.e. resolve it in the current context.

user=> (def five 5) ; create a named ref representing the number 5
#'user/five
user=> five ; five will yeild its internal value
5
user=> `five ; syntax quoting five will fully resolve the SYMBOL
user/five
user=> `~five ; within a syntax quoted block ~ wil resolve the value in the current context
5

This forms the meat and potatoes of creating macros which are, to be highly reductionist, functions that return blocks of syntax with parts evaluated in varying contexts.

`~@` - Unquote splicing macro

See ` (syntax quote) and ~ (unquote) for additional information.

~@ is unquote-splicing. Where unquote (~) deals with single values (or treats its attached item as a single item) ~@ works on lists and expands them out into multiple statements. Think JavaScripts .apply method that takes an array and expands it out as arguments to the applied function.

user=> (def three-and-four (list 3 4))
#'user/three-and-four
user=> `(1 ~three-and-four) ; treates as a single statement produces a nested list
(1 (3 4))
user=> `(1 ~@three-and-four) ; expand out as seperate statements
(1 3 4)

Again this gives us a lot of power in macros.

` - Syntax quote

See ~@ (unquote splicing) and ~ (unquote) for additional information.

` is the syntax quote. When used on a symbol it resolves the symbol in the current context,

user=> (def five 5)
#'user/five
user=> `five
user/five

When used with lists (remember every thing in Clojure is data) it forms a template for the data structure and won't immediately resolve it.

user=> (1 2 3)
ClassCastException java.lang.Long cannot be cast to clojure.lang.IFn  user/eval832 (NO_SOURCE_FILE:1)
user=> `(1 2 3)
(1 2 3)

You'll see this most often in the context of macros. We can write one now,

user=> (defmacro debug [body]
  #_=>   `(let [val# ~body]
  #_=>      (println "DEBUG: " val#)
  #_=>      val#))
#'user/debug
user=> (debug (+ 2 2))
DEBUG:  4
4

Code updated based on recommendations from Leif Foged

The macro takes a single statement wraps it in a quoted let block, evaluates and prints the result and then evaluates the body. In effect this defmacro call returns a quoted data structure representing the program we are writing with it. The ` allows this to happen.

`var-name` - Earmuffs

Earmuffs (a pair of asterisk bookending var names) is a naming convention in many LISPs used to denote special vars. Most commonly in Clojure this seems to be used to denote dynamic vars i.e. ones that can change depending on where you are in the program. The earmuffs act as a warning that "here be dragons" and to never assume the state of the var. Remember this is a convention not a rule.

Core Clojure examples are *out* and *in* which represent the standard in and out Writers for Clojure.

`>!!`, `<!!`, `>!` & `<!` - core.async channel macros

These symbols are channel operations in core.async - a Clojure/ClojureScript library for channel based asynchronous programming (specifically CSP - Communicating Sequential Processes).

If you imagine, for the sake of argument, a channel is a bit like a queue that things can put stuff on and take stuff off then these symbols support that simple API.

>!! & <!! are blocking put and take respectively
>! & <! are, simply, put and take

The difference being the blocking versions operate outside go blocks and block the thread they operate on.

user=> (def my-channel (chan 10)) ; create a channel
user=> (>!! my-channel "hello")   ; put stuff on the channel
user=> (println (<!! my-channel)) ; take stuff off the channel
hello

The non-blocking versions need to be executed within a go block otherwise they'll throw an exception

user=> (def c (chan))
#'user/c
user=> (>! c "nope")
AssertionError Assert failed: >! used not in (go ...) block
nil  clojure.core.async/>! (async.clj:123)

While the difference between these is well outside the scope of this article fundamentally the go blocks operate and manage their own resources pausing execution of code without blocking threads. This makes asynchronously executed code appear to be synchronous and removing the pain of managing asynchronous code from the code base.

`<symbol>?` - Predicate Marker

Putting ? at the end of a symbol is a naming convention common across many languages that support special characters in their symbol names It is used to indicate the thing is a predicate i.e. that it poses a question. For example imagine using an API that dealt with buffer manipulation

(def my-buffer (buffers/create-buffer [1 2 3]))
(buffers/empty my-buffer)

At a glance how would you know if the method empty in this case,

Returned true if the passed in buffer was empty, or,
Cleared the buffer

While the author could have renamed empty to is-empty the richness of symbol naming in Clojure allows us to express intent more symbolically.

(def my-buffer (buffers/create-buffer [1 2 3]))
(buffers/empty? my-buffer)
false

This is simply a recommended convention not a requirement

Clojure Style Guide

`<symbol>!` - Unsafe Operations

The Clojure style guide has this to say

The names of functions/macros that are not safe in STM transactions should end with an exclamation mark (e.g. reset!).

You'll most commonly see this appended to function names whose purpose is to mutate state e.g connecting to a data store, updating an atom or closing a file stream.

user=> (def my-stateful-thing (atom 0))
#'user/my-stateful-thing
user=> (swap! my-stateful-thing inc)
1
user=> @my-stateful-thing
1

This is simply a recommended convention not a requirement

Clojure Style Guide

`_` - Irrelevant var

When you see this used as function arguments or similar it is a common naming convention for vars or arguments you are not interested in using. That is you don't intend to use them so you aren't really interested in thinking of a useful name for them.

This is an example using the add-watch function that can be used to add callback style behavior when atoms change value. Imagine, given an atom, we want to print the new value everytime it changes,

(def value (atom 0))

(add-watch value nil (fn [_ _ _ new-value]
                       (println new-value))

(reset! value 6)
; prints 6
(reset! value 9)
; prints 9

add-watch takes 4 arguments but in our case we only really care about the last argument - the new value of the atom. I don't really want to spend time thinking of names for these arguments I'll never use, nor do I want to generate a long line of text unnecessarily (of course I could have used the shorthand #(println %4) but that defeats the purpose of this example).

Many thanks to everyone who has contributed ideas and [the copious amounts of] spelling corrections (crikey I'm bad at speelingz - so thanks Michael R. Mayne, lobsang_ludd). I've tried to call out people who have specifically asked for things. Sorry if I've missed you.

A Brief Overview of BaseX

Sat May 17 00:00:00 UTC 2014

There are technologies that I have used to solve problems that just make me happy while using them. They make solving the problem so easy that I don't have to burn time getting them to work or try bending them to my needs.

BaseX, an XML database, is one of those. Yes it's an XML database, yes XML is, like, totally lame and all but trust me, it exists and there are contexts it works.

So I wanted to write about BaseX because while it may not be shiny it makes up for it in pure practicality. One day you may need to interrogate XML and maybe BaseX will save your sanity as it did mine. Also XQuery is rather wonderful.

BaseX

BaseX is an XML database written in Java. That may sound like "the worst" but let me tell you why it is rather the opposite.

So yeah it's cool because it's NoSQL. It stores XML documents so it's doubly cool because it's a document store. Not only that but it doesn't care about DTDs and XML schemas so, as long as they are well formed, it can store any XML document. That makes it triply cool because it's schemaless. Heavens, it's basically MongoDB for XML. Well not quite ¹ - the design goals are not the same.

So what is BaseX useful for? Well one of the areas that it shines is Open Data. There is a lot of publicly available data locked up in static XML documents.

One example is IATI - the International Aid Transparency Initiative. IATI is many things,

an organisation that promotes transparency of foreign aid spend,
an XML standard for publishing data consistently, and,
a registry to access this open data

Many organisations and governments that spend money on foreign aid publish data via IATI. Data is usually published as many XML files and according to the registry there are 241 organisations publishing 3,343 datasets (XML files). Thats gigabytes of data. Not exactly big data but still a respectable volume of XML.

To top it all off the registry only provides a centralised lookup for datasources, pointers to XML files, via metadata. It does not extract the data from within the XML files. So the data itself is still locked up in static XML files spread across the globe.

None of this is specific to IATI. Like it or not there is a lot of data exactly like this.

Now imagine you want to ask a question of this type of distributed XML data and you've discovered the sweet spot for BaseX. You feed a BaseX database a batch of XML files, it indexes the contents and allows you to query over the top of it as if it was a single root document.

Querying

If there is one thing that can be said about XML - it's really well thought out. Yeah yeah there are aspects of the ecosystem that could be described as over thought but as a data format its not half bad. Thanks to this "thoughtout-ed-ness" BaseX doesn't need to invent any new concepts to support querying. XML has already got its own query language, well actually two, XPath and XQuery. Both are mature, well documented, well understood query languages.

XPath

XPath is like the CSS selectors of the XML world. A bit more verbose than CSS selector syntax but essentially the same idea. In fact there are some features, such as selecting nodes based on conditions of their children, that CSS selectors could do with. Take this rather contrived example,

/repository/projects/project[@title="My Blog"]/releases/release[@type="beta"]/release-date/text()

This would return the release-date of all the beta releases of the project titled My Blog. I did say contrived didn't I? So XPath is useful for selecting things but not manipulating or transforming anything.

XQuery

XQuery is useful when you want to do more than just select nodes in a document. In fact XQuery is kind of an amazing language. Seriously. Its based on a SQL-like expression set called FLWOR (FOR, LET, WHERE, ORDER BY and RETURN) that when mixed with XPath you have a nice functional language. Everything is a sequence making it conceptually rather LISPy yet the syntax is much like ML.

Here is a slightly more complex, real life authenti-genuine, example featuring list manipulation, dynamic XPath execution, external parameter passing and other typical XQuery stuff,

declare variable $start external;
declare variable $limit external;
declare variable $reporting-org external;

let $filter := "$db//iati-activity"
let $filter := concat($filter,
  switch($reporting-org)
    case "" return ""
    default return "[reporting-org/@ref=$reporting-org]")

let $db         := db:open('iati')
let $bindings   := map { '$db' := $db,  '$reporting-org' := $reporting-org }
let $activities := for $activity in xquery:eval($filter, $bindings)
           order by $activity/iati-identifier
           return $activity
return <result>
  <ok>true</ok>
  <iati-activities generated-datetime="{ current-dateTime() }">
    <query>
      <total-count>{ count($activities) }</total-count>
      <start>{ $start }</start>
      <limit>{ $limit }</limit>
    </query>
    { subsequence($activities, $start, $limit) }
  </iati-activities>
</result>

Clients

I've already said BaseX is written, unsuprisingly, in Java but as with most data stores use is not limited to the JVM. BaseX runs as a server that you send commands to over TCP and, as you'd expect, there are many clients available to abstract this lower level communication.

I've used the Java, Scala and Ruby client and they are all pretty much the same. I'm also currently in the process of building a Clojure based one that wraps the standard Java client. Again the API is very similar to the other clients.

BaseX doesn't have an extensive feature set and so the clients aren't exactly complicated.

The clients follow a similar pattern of having two modes. A standard mode that provides functionality for connecting to a server and sending arbitrary commands and a query mode that allows you to create a query on the server, bind external values to the query and stream the results to the client. You can think of queries in query mode as similar to SQLs PreparedStatements.

In reality query mode is just a bit of rich sugar to make the common task of querying a bit less bloaty and there isn't anything in it you can't achieve in standard mode - its all just sending a basic set of commands over a socket. The power lives within the server.

Some clients (Java, Scala, C# and Clojure for example) also support the notion of events. With these clients you can ask sessions to watch and unwatch a special database type of event. The typical signature for watch looks like this,

session.watch("my-event", new EventNotifier() {
  @Override
  public void notify(final String value) {
    System.out.println(value);
  }
});

On the database side you need to have created an event called my-event for this to succeed

CREATE EVENT my-event

Then at some point you can fire an event and pass values to it.

db:event("my-event", "1 to 4")

Each event can be passed an XQuery expression that will be evaluated and passed to any registered handlers on the client side. This can be one or more opened session as long as they have watched the named event.

Modules

BaseX can be extended via modules too. These can add extra features to the database itself as well as enhance XQuery itself. There are a heap of ready-rolled modules for all your needs. Just to call a few out,

JSON Module - extend XQuery to support reading and writing JSON. Yep JSON in and out of an XML database. Bananas.
Higher Order Functions Module - get some monads in your XQuery
Full Text Module - Every database needs full text search capabilities these days it seems and BaseX need not be an exception.
Unit Testing Module - Testing is dead but don't let that hold you back.
Geospatial Module - another must have trend in databases these days.

There are a heap of them allowing you to customise BaseX to your needs.

Other Notable Things

There are a few other things worth mentioning that don't fit into their own category.

Admin GUI

One of the things I miss with a lot of the NoSQL databases is a good Admin app. These apps let me poke around the data in my database, model solution and generally visualise all those bits and bytes.

SQL Server got this bit right. SQL Server Management Studio is a really handy tool for understanding your data and its container a bit better.

BaseX has you covered. It has a rather nice client that lets you write XPath/XQuery, manage documents and visualise not only the data but also profiles queries and shows how they can be optimised.

Its written in some Java based desktop technology so the UI itself is a bit basic but given what you can do with it, thats more than forgiven.

Binary Data

BaseX lets to read and write binary data as well. Being binary you can't do much more than read and write it but it means that you can, if needs be, store binary blobs alongside your XML.

Scaling

At one point I was using BaseX to query across a large amount of XML data. Not a huge amount but large, about 13GB worth. The query I was running, while it wasn't basic wasn't exactly complicated either but it did result in a large result set and the query times where not suitable for real time applications (single queries running 3-6 seconds). In my case that was fine. I suspect there are very few cases where you'd genuinely need real time responses from a large blob of XML but still.

BaseX doesn't provide any sort of answer for distributed environments, you'd need to provide some sort of basic sharding strategy yourself if that is what you are after. Perhaps if someone gets drunk enough they'll write some sort of Hadoop integration for offline querying of massive XML datasets. Perhaps.

In a Nutshell

BaseX fulfils a need. Not a need many people will have but a need nonetheless. It does it well and it does it without inventing a new query language or offering 1000 shiny features. It uses XQuery which is, quite honestly, a wonderful language to work with. It's functional, minimal, expressive and powerful. The Admin GUI offers some nice insight into your databases and the clients offered across most common and current languages make working with BaseX straightforward.

While I am on this subject BaseX is of course not the only XML database. You've got others like eXist, MarkLogic and Sedna. I've never ventured into other systems (save for a brief look at eXist) so can't comment on them but in terms of feature set and licensing BaseX looks to be one of the better implementations.

^{1: I'd imagine, though, if a Java Certified Enterprise Architect decided to write a document store from scratch they'd probably end up with an XML database har! har!.}

Clojure Cookbook

Thu May 15 00:00:00 UTC 2014

Broad. If I had to describe the Clojure Cookbook in a single word broad would be it. If you're looking for in-depth explanations of all the minutiae and edge cases of a particular topic in Clojure this book is probably not going to fit the bill. If, on the other hand, you're spelunking through the Clojure landscape building apps and generally tinkering with everything you get your hands on then this book is the perfect companion.

What is it?

The Clojure Cookbook, as the name suggests, is a collection of recipes (short-ish articles) for achieving various things with Clojure. Recipes cover a broad range of topics. You'll encounter string manipulation, writing web apps, Datomic, testing with Midje, Cascalog and a bunch of other things. On quick calculation there are about 169 recipes in the book from about 60 contributors, all reviewed and curated by the books authors.

Who is it for?

Good question. A broad book is going to have broad appeal. If you're starting on your Clojure journey you will initially find usefulness in the more low level recipes in the book and eventually graduate to the more general higher level recipes such as writing web apps and accessing databases. But if you've been working with Clojure for some time there is always going to be some aspect or task you've not had to deal with yet and again many recipes in this book could prove useful. At the very least they could act as refreshers for the tasks you only need to approach once in a while.

Another way to consider using this book is as inspiration. If you're wanting to take your Clojure experience to the next level and venture into new areas you could use this book as a sort of guide. This is where I see the charm in this type of book rather than the function. In my downtime I like flicking open the contents and picking a recipe at random and having a play with whatever technique or technology it's focused on.

That said if you like to get down to the nuts and bolts of a topic and really dig into it you're going to need another way to do that. The recipes in this book usually offer some insight into the rationale behind them (they "why?" if you will) but given their terse nature you could be left with outstanding questions. Also, the level of detail in each recipe varies e.g. you can expect more discussion around an article covering, say, string manipulation than you will about broader topics like distributed computing. That said you are never left wondering why you'd actually do the thing you are reading about - it's either obvious or discussed.

Worth it?

Yeah. Listen - you don't need to buy this book. Its available for free online and a quick Googling could often lead you to articles on the same topics but lets think about this a little differently. Books like this are the result of community collaboration, of people coming together and producing a body of work that deserves a place on your bookshelf. So, without getting to idealistic, not only are you enriching yourself but you are celebrating community and I value that.

As a body of work - online or off - whether you are diving into Clojure for the first time or a seasoned veteran about to venture into a new area of the ecosystem there is something in this book you can get value from.

I was fortunate enough to receive a complimentary copy of the Clojure Cookbook but I was in no way contracted to write a review. I do so simply because it's the least I could do to promote a book I find useful.

On onpaste

Thu May 08 00:00:00 UTC 2014

When they say,

We'd lose our security certificate if we allowed pasting [into input fields]. It could leave us open to a "brute force" attack.

They forget that their webpage is simply a client of their backend service. There are many other existing ones.

They forget that stopping "brute force" attacks is a matter for their back end.

They forget that JavaScript can be disabled, fail to load or used to turn off 'onpaste'.

They forget that these practices force their customers to use simpler passwords

They forget that their actions make their customers more vulnerable.

They forget that when a user, who was for all intents and purposes forced by them to use a simple guessable password, has their account compromised they use words that put the blame on that user and make them feel bad.

I say forget but I really mean ignore the fact. I guess it's easier to hide behind outdated words and policy than understand the needs of the user, your customers.

Managing Environment Variables in Clojure

Tue Apr 29 00:00:00 UTC 2014

If you've never read the Twelve Factor App methodology I strongly recommend you do, it's a great guide born from real life experience of building many, many web applications.

One of the rules in the Twelve Factor App methodology is that you "Store config in the environment" or, more specifically, store config that is likely to change across environments in the environment. It makes sense as it ties the target environment and its necessary configuration together instead of having to juggle if ENV=test|dev style flags around your code base or swap out different config files per build.

One other positive side effect of this approach is that secure configuration doesn't accidentally get checked into source. Every one needs database credentials or client secrets but no-one want to share them with the world.

Supporting this approach in Clojure is fairly simple and there are a few ways to do it.

System/getenv

Clojure sits on the JVM and the JVM has the System namespace which provides utilities and classes for accessing the running systems features. System.getenv and System.getenv(String) give us access to environment variables from Clojure.

(System/getenv "DATABASE_URL")
;=> http://user:pass@dburl/mydb

We can also provide a fallback for environment variables that don't exist

(or (System/getenv "NOEXIST_DATABASE_URL") 
    "http://localhost/mydb")
;=> http://localhost/mydb

In this case NOEXIST_DATABASE_URL isn't set as an environment variable so we fallback to our hardcoded version.

This isn't uncommon and often its done to support development environments. For example Heroku (where the original 12 factor app methodology arose) add environment variables for any addons you may activate such as DATABASE_URL for the Postgres DB addon. So rather than write different handlers per environment many people would either.

Use the fallback URL to represent their development environment
Add the environment variable to their shell

Both of the approaches have problems. Using a fallback URL can lead to problems where an environment isn't correctly configured and starts using the wrong database. If it can write to it there will be trouble, if it can't you'll be scratching your head wondering whats going wrong for longer then necessary. Setting a project specific environment variable in your shell can lead to pain when things get overwritten across projects and pollution if you're setting it in an automated fashion (.bashrc) or you'll simply forget to set it per use when doing it manually.

weavejester/environ

A neater solution is available in the form of environ - a simple library for managing environment variables. The clever thing environ does is merge environment variables from multiple sources into a single map.

So by adding it to our project.clj

[environ "0.5.0"]

We can import and use it where we need,

(require [environ.core :refer [env]])

(env :database-url)
;=> http://localhost/mydb

So how are the environment variables sourced? Well environ looks in a number of areas in order,

A .lein-env file in the project directory
Environment variables
Java system properties

It also keywordises the variable names as you can see from the example above.

Great, so why is this more useful? The big benefit I've discovered is the .lein-env file in the root of my project. This file holds a map of environment values that can be useful during development. There are 2 ways to create this file.

The RIGHT way as set out by the creator of the project, and,
The WRONG way as in how I use it

environ also has a plugin available lein-environ which sucks profile specific settings from ~/.lein/profiles.clj and/or a project specific profiles.clj (which should not be checked into source) and creates the .lein-env file when Leiningen does its thing (effectively making .lein-env a transient file). A .lein-env file looks a little something like this.

{ :env { :database-url "http://localhost/mydb" 
         :client-token "QWERTY12345" } }

Caveat: Now you can create this manually (this is what I have done for a small project I'm working on) but it's not officially recommended for a few reasons. Firstly the minute you add the lein-environ plugin to the project your important setting will get wiped out and secondly you lose out on the ability to vary setting across profiles as well that you'd get from profiles.clj.

Big thanks to @weavejester the creator of environ for the tip off about my dubious use of .lein-env

So back to .lein-env - You end up with the .lein-env file on your machine with development (or profile) specific values and you forget about fallbacks and setting environment variables and the potential conflicts with other projects. Then on your production and test systems you can source values from actual environment variables without having to change your strategy or use (if (= "test" (:env config))) style checks.

Its worth noting that .lein-env is already added as a match in the default .gitignore from lein new ... so wont be checked into git - BUUUUUUT... if you do manage to check in and push your .lein-env file (or any project specific profiles.clj file) and it does happen to contain secure tokens and passwords (hashed or otherwise) you must assume those values are already compromised even if you manage to purge them from history.

Play! 2 for Scala (Pluralsight)

Tue Apr 15 00:00:00 UTC 2014

Why Scala? Because C#

Thu Apr 03 00:00:00 UTC 2014

Strange as it may sound this isn't a post about promoting the adoption of Scala but if its a conversation you're already having there may be some points here that might help. I often hear rationale for Scala over Java based around the functional style, immutability and slightly better concurrency model baked in but I've always found there is a simpler point to be made. Time will tell wether the recent release of Java 8 will add enough to old faithful to slow the adoption of Scala.

I often get asked by people I'm introducing to Scala, "Why Scala and why not Java" and more often that not the topic turns to comparisons between Scala and C#. Its not just me either. Last year I was involved in a few rounds of procurement, interviewing prospective IT companies and team and a common pattern that came out of those interviews was people would be offering us teams composed of their C# developers over their Java developers. Scala experience was, at that time, still a bit light on the ground and yes you could argue that the general trend toward fully open source stacks in many markets was squeezing the amount of available work for MS folks but the message most people delivered was their C# developers were generally more capable of transitioning to Scala than their Java people. I can't necessarily dispute this either because I certainly feel my background in C# helped me ramp up on my first Scala project much faster than my Java experience had.

Allow me a little example (ripped right out of my Scala for C# Developers talk)

"CHRISTMAS".makeChristmasy
// => ***CHRISTMAS***

There are three important things to know here

The String class in Java is final and can't be extended.
The String class in Scala is final and can't be extended (it is the Java String class)
The String class in C# is sealed and can't be extended

Yet we just called a function on it that isn't part of Strings signature (it's common knowledge Scala has a lot of bells and whistles but trust me the String class doesn't normally have a makeChristmasy method)

To implement this we can make use of Scalas Implicit Classes. We can write a class that wraps a class, String in our case,

implicit class FancyString(val s: String) { 
 def makeChristmasy = "***" + s + "***" 
}

Import this into our namespace and voila our String variables and values have the makeChristmasy method available to them. Under the hood the Scala compiler is auto wrapping the String instances with FancyString where necessary. Kinda like this,

new FancyString("CHRISTMAS").makeChristmasy

Sure its essentially a bit of syntactic sugar but it can, when used wisely, help make code clearer.

C# can do this too. C# has extension methods,

public static class FancyString
{
    public static int MakeChristmasy(this String s)
    {
        return "***" + s + "***";
    }
}

So what? Scala has a feature that C# has and Java doesn't. Hell lets do the same in Ruby,

class String  
  def make_christmasy
    "***#{self}***""
  end  
end

So now can Ruby developers transition to Scala faster than Java devs? No, well maybe I'm not sure, stop confusing me, besides open classes in Ruby are a vastly different and scarier thing than extension methods and implicit classes.

Your point being...?

There is a certain power and flexibiltiy that a feature like implicits offer. An ability to coerce the underlying language into something that makes more sense in the given context. Yeah its open for abuse but we're supposed to be grown ups here people. Lets not force people to use plastic knives to cut steak just in case they stab themselves. Of course its not just implicits - the language is simply a richer language.

What Scala offers is somewhat like C# on the JVM. Java fell behind C# a long time ago and even with the release of Java 8 it is still not quite there. There is a degree of expressiveness in C# that enabled me in my .NET days to produce better, more grokable systems in C# than anything I produced in Java. I found the same with Scala. Sure there are many, many dark corners in Scala but they can be avoided be that through a common understanding, peer review, automated checks like Sonar or even through the use of Scalas in-progress ability to add/remove language features through modular design. "Scala: The Good Parts" as Thoughtworks termed it in their tech radar is a theoretical subset of the language that offers better collection support, terser syntax and a reduction in the need for pattern boilerplate without having to upgrade your JVM.

So while the gap between Java and Scala has lessened it's still there and given the cycle time for new Java releases it's not going to shorten any time soon. Perhaps you think there are times when you want your language to be a constraint on your team. I'd be suprised if you've never encountered a room full of mad language scientists trying to golf some operation down to 3 convoluted lines because "FUNCTIONAL PROGRAMMING YAY". But imposing artificial constraints because of a team problem is just going to frustrate the "good" people and frankly instead of an unmaintainable mess due to bonkers code golf you'll probably get an unmaintainable mess due to over application of EVERY PATTERN BECAUSE OBJECTS!

So why Scala? Because there is a core feature set in the language that enables people to produce nice clean straightforward code with less boilerplate and enough power and flexibility to make the intent of the resulting code clearer. Like C#.

Experience Report: Migrating from Middleman to Stasis

Tue Apr 01 00:00:00 UTC 2014

I recently migrated my statically generated site (yep this site) from a Ruby/Middleman solution to a Clojure/Stasis based solution. This post is a kind of experience report on the migration process. It isn't a guide for using Stasis and if that is something you're looking for I can't recommend Christian Johansens "Building static sites with Clojure" post enough.

Background

My site has transitioned through various platforms - from WordPress to Posterous, Jekyll and Middleman. Its also went through a few types - Internal corporate blog, public blog and finally a site that represents my role as a contractor.

I never really enjoyed WordPress or Posterous because there was a distinct lack of control and way too many features and points of failure and while Jekyll gave me flexibility its pattern for extension was rather undocumented and ambiguious. Middleman offered me the flexibility with a very simple programming model.

The next migration, the one I'm covering now, came about because the site had accrued a lot of content debt over the years and I needed to tidy it up. Problems such as

duplicated and redundant YAML frontmatter settings and logic,
a lot of different file types
the migrated Posterous stuff was locked up in prerendered HTML
many of the older posts use embedded gists and I wasn't happy with that

Decision to Migrate

So why move from Ruby to Clojure, Middleman to Stasis? Truth be told it was initially a learning experience. I was knee deep in using Clojure and decided the project was substantial enough to help further my education. I had started with refactoring the existing Ruby codebase but the content tidy up effort was proving awkward in Ruby. Once I started the work in Clojure it was clear I was able to better express my intent.

Observations

The next few sections are observations I've made during the migration.

Stasis gives you less (and that's OK)

Statis just offers a few functions that are useful when creating static web sites.
No more. There are no batteries included.

Thats directly from the Stasis README and as you can guess it's spot on. Fundamentally speaking Stasis gives you two entry points

A ring handler for serving content
An export-pages function for saving content to disk

Both of these expect a map of path and content and thats it. You could serve an simple hello world style site like this.

(stasis.core/export-pages { "/index.html" "<html><body>Hello World</body></html>" }
                          "build")

Running that would result in and index.html file in the /build folder of your project with the contents above.

To complement this stasis also gives you a method to slurp a directory, read its contents and generate one of these content maps. This is an excellent starting point for transforming file based content.

In my site I slurp a directory of metadata enriched markdown files (YAML frontmatter) and thread them through a bunch of transformation functions. The heart of my content processing is a simple little function

(defn entries
  []
  (let [entries (slurp-content "resources/content/entries")]
    (->> entries
         (filter :published)
         (map #(assoc % :uri
                      (case (keyword (:type %))
                        :article    (article-uri %)
                        :screencast (screencast-uri %)
                        :talk       (:url %)
                        :project    (:url %))))
         (sort-by :date)
         (reverse))))

slurp-content in this function extracts the file contents and converts it to a map based on the YAML frontmatter and the content body

At the end each bit of content (article, screencast, talk entry etc.) is a map that can be passed around to generate category pages, RSS feeds, HTML pages. Content as data so to speak. So in future if I want to provide a JSONified version of all talk metadata I can just add another function that merges my new paths into the site map that Stasis uses.

This sort of work was mostly handled by Middleman internally and transformations configured by providing different file extensions to the content (which could be stacked like .html.erb.md). This meant that behaviour was controlled by manipulating content rather than it being treated as data. In Stasis this means there is a bit more code involved but the end result is perfectly acceptable.

There is a simplicity in Stasis I could never achieve with Middleman.

Expressiveness of Clojure

When it came to tidying up the content I found the use of an interactive REPL and a few simple libraries meant I was able to achieve things quickly and easily. When working with files combining the REPL with Git gave me a transactional approach to messing with files without fear. I've saved most of my efforts in the fom of a gist for this work.

One of the biggest challenges I came up against was taking the old articles that contained embedded gists and replacing these with syntax highlighted code blocks. This involved

Scraping the script tags that linked to embedded gists from the articles
Using the Github API to load the content and file type of each Gist (some had multiple files)
Run the content through the pygments based syntax highlighter to produce a code block
Swap the script and potential noscript block from the offending files
Write the new content out.

It sounds easy on paper (and maybe for you it is) but previous attempts in Ruby had driven me to give up. With Clojure and Enlive this work was easier to express and achieve. Of course it wasn't without problems. I found Enlive confusing at times but the interactive nature of the REPL meant I could break the problem down and isolate pain points.

Clojures REPL is more powerful than irb and this allowed me to babystep my problems. Clojures syntax, simplicity and powerful standard lib allowed me to express my intent more clearly, with less code and fewer problems. The two together gave me an environment that allowed me to get to where I wanted with fewer frustrations.

Generation Time

I build my site with an alias I set up lein build-site. Previously I used bundle exec middleman build. At first I noticed that the time to generate the static site in Clojure felt a lot longer. In fact the generation process itself is probably a bit slower. But, there is always a but, overall a fresh build and deploy (thanks to Travis) actually takes a tiny bit less (a few seconds difference, nothing to write home about).

It seems a lot of the build time in Ruby comes from the resolution of dependencies of which Middleman has a lot more than Stasis. Note: This analysis is by no means in-depth of course and ultimatley it still only takes about 3 minutes from commiting a chance to seeing it live (depending on the speed of Travis picking it up).

Credits

I've previously mentioned Christian Johansens "Building static sites with Clojure" post and I'd like to acknowledge that it was the post that finally made me decide to migrate. His post was invaluable during the inital migration (especially around the syntax highlighting).

While the post kicked of the decision to migrate the following list of technolgies made that transition possible.

Stasis - provides the core of the site generation
Optimus - Static asset optimisation (bundling, minification etc.)
Enlive - DOM manipulation library that was an essential part of unembedding the gists and replaceing code blocks with syntax highlighted blocks.
cegdown - Markdown processing library. Essentially a Clojure wrapper for Pegdown
clj-yaml - Encoding and decoding of YAML frontmatter
fs - A bunch of file system utilities. Used a lot when renaming and replating content files.
slugger - Genreates HTML friendly slugs for articles. This is a port of the stringex Ruby library used in Middleman which was essential to avoid breaking original urls
clygments - A Clojure wrapper around then Pygments syntax highlighting library.

flic - Simple feature toggles

Thu Mar 13 00:00:00 UTC 2014

Defocus

Tue Mar 04 00:00:00 UTC 2014

Let me tell you a story. It's one of those stories you look back on and laugh at how ridiculously clueless you had been. It's not a great story, it's a bit silly in fact.

In my first year of my Software Engineering course at university we were introduced to a number of languages including but not entirely limited to Java, Ada95, Pascal, Prolog, Standard ML and JavaScript. The intent behind this exposure to so many languages in one go was to get us to understand the various language styles such as object orineted, logic and functional. My previous experience of languages was limted to breaking everything in JavaScript, Pascal, Visual Basic and its subset VBA. It wasn't easy, jumping between styles and syntax and concepts. But you keep going and eventually you get it. So eventually I got it. Just in time to move to the next language.

In particular I really enjoyed working with ML, I still own Elements of ML Programming with it's bizarre front cover resembling a treasure map. The functional approach it used, and the power of this appraoch for building terse solutions was a breath of fresh air compared to the Java 1.2 stuff we were using on another course. I enjoyed it so much I wanted to do well in it and so would work through the course material, the optional material and then some other stuff during the practical lectures. Nothing big, just little functions that did factorials and the usual stuff.

One day I was going through the usual exercises determined to get the mandatory exercises done and crack on with the optional ones (usually more interesting). About an hour and a half in I was trying to write a function that given a number would return the number if it was less than 10 otherwise it would return 1. Simple, right? I got about this far.

(* do some number adjustment magic *)
fun adjust number = if number < 10
                        then number
                        else <???????>;

adjust 3; (* expects 3 *)
adjust 10;(* expects 1 *)

I'm head down learning new things, trying to understand the syntax of ML and I'm stumped because I don't know how to return 1 from a function. I thought and thought and thought about it. Nothing. Eventually I remembered dividing a number by itself yield 1. Pure. Unadulterated. Genius.

fun adjust number = if number < 10
                        then number
                        else number/number;

But nooooooooo. Running this gives us the following error,

Standard ML of New Jersey v110.74 [built: Fri Jan 24 17:53:58 2000]
- stdIn:4.30-4.43 Error: operator and operand don't agree [literal]
  operator domain: real * real
  operand:         int * int
  in expression:
    number / number
-

My ML-fu was still weak and this had me stumped. I tried various tweaks of the same thing but I could not get this thing to return a 1. Defeated by a dumb number I left the class early and went for a walk.

About 10 minutes into the walk I'd forgotten about the class and the absolutely mind bending puzzle of returning 1 from a function. Hmmmmm Returning 1. Just return 1. else 1. Wait, wait you can just return things. 1 is a thing. Return the thing that is 1. The thing that is 1 is 1. Return 1.

Ffffffffffffffffffffffffffffffffffffffacepalm

It only took about 11 or 12 minutes of extracting myself out of that narrow focus I was in. A focus so narrow that fundamental programming concepts such as literals weren't even in my periphery. I simply stopped thinking correctly. I could have sat there until the class finished and burnt more time.

Of course I'm not saying anything new here. I'm recounting a 14 year old story that still happens to me today. But I think now I've matured enough to know that my continued effort is becoming net negative and to walk away. It feels like common sense now and you may think the same thing. But its a point worth reminding yourself of sometimes.

Focusing on a single point for too long will blind you to alternatives. Take time to reflect, take time to think AWAY from the problem. Use a shower, hammock or long walk. Do some exercise. Talk to a third party such as a rubber duck or a real person. Ask yourself "how would I do this in something I'm familiar with?".

Defocus.

Building Command Line Apps with Clojure

Sun Mar 02 00:00:00 UTC 2014

I recently read an article by Ryan Neufeld around parsing command line arguments with Clojure. It's a good article and you should read it if the idea of building command line utilities in Clojure floats your boat.

To complement that article I've decided to take the building of the command line app one step further and support running your creation as a standalone command. Of course its possible, as demonstrated in the article, to run the app using Leiningen directly but you typically want to be bundling your app into a self conatined package (minus any necessary runtimes - in our case the JVM) for deployment to other environments.

So lets start were Ryans article left off, a simple command line app that can be run via Leiningen. The core.clj looks like this,

(ns runs.core
  (:require [clojure.tools.cli :refer [cli]])
  (:gen-class))


(defn -main [& args]
  (let [[opts args banner] (cli args
                                ["-h" "--help" "Print this help"
                                 :default false :flag true])]
    (when (:help opts)
      (println banner))))

And the project.clj looks like this

(defproject runs "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.5.1"]
                 [org.clojure/tools.cli "0.2.4"]]
  :main runs.core)

As Ryans article mentions we can run this via lein run and pass our arguments in.

lein run -- -h

Uberjar

First thing we want to do is take Leiningen out of the equation and bundle the app as a standalone JAR. Unlike other JVM build tools that need to be extended with plugins such as shade, onejar and assembly Leingingen comes with uberjar prebundled. To create a basic uberjar, a JAR containting your application code and all the necessary dependencies, we can call lein uberjar. There are also a bunch of configuration options that go along with uberjar but what we have specified is sufficient to build our jar. The JAR will be output to target dir appended with -standalone. We can run this JAR using via the usual Java way

java -jar target/app-0.1.0-SNAPSHOT-standalone.jar -h

lein-bin

I don't know about you but I'm lazy and I like my commands names to represent what they do (roughly) so the thought of running java -jar blahblah.jar -h every time I want to run the command is just "meh". We could write a shell script to run the more verbose command but thats less than ideal (it's messy and on top of my laziness I'm a tad OCD about carting around wrapper scripts).

Enter lein-bin a Leiningen plugin that utilises the fact ZIP files and therefore JARs can have stuff prepended to the front of it. What it does is prepend shell commands to your JAR making it executable in and of itself.

So to use lein-bin we need to add the plugin to our projects project.clj or the global Leiningen profile (~/.lein/profiles.clj)

{ :plugins [[lein-bin "0.3.4"]] }

We can also add some lein-bin related configuration like changing the output name (there are some other options as well, all noted in the lein-bin README)

:bin { :name "runs" })

So running lein bin now leaves us with another file in target called runs that is executable.

target/runs -h

Now we've got a command line util called runs (yeah a terrible name) rather than some verbose Java command or being force to use a build tool on all environments. It's the little things.

SPAs and Enabling CORS in Spark

Thu Feb 20 00:00:00 UTC 2014

Having been tinkering with Spark again I needed to build a simple bit of infrastructure for building single page applications that could be deployed anywhere. I also needed to potentially support CORS (Cross Origin Resource Sharing) so that remote client-side JavaScript applications could actually access the JSON API aspect of the solution. Doing this with Spark is pretty simple but I thought I'd document the process to avoid some people (or future me) having to answer the same questions again. I'm going to assume you know a bit about Spark, if not have a skim of the README on the site.

Building a SPA

An SPA or Single Page Application is an application that serves a single HTML page and included JavaScript on that page is responsible for dealing with talking to the server, getting data, rendering different views and handling view navigation. This is different from the traditional approach of rendering views on the server through the typical request/response lifecycle. I'm not going to debate the merits and drawbacks of either strategy in this post let's just assume that you've decided to go with an SPA in a Java context and you want something simple. I'm also not going to suggest any sort of front-end framework here - that'll depend on what you need, right? So a sample anatomy of an SPA could look like this,

The root path at http://website/ returns a static index.html file
The index.html file loads the necessary static assets such as JavaScript and CSS
The JavaScript calls local API endpoints to load data ¹ and render content through client side templates or straightforward DOM manipulation

That's it. So in Spark to support these needs you need to do a few things,

The API endpoints

JavaScript loves JSON, it makes sense. So we need to create an API endpoint that returns JSON responses.

get(new JsonRoute("/api") {
    @Override
    public Object handle(Request request, Response response) {
        return new HashMap<String, Object>() {{
            put("message", "Hello World");
        }};
    }
});

This stubbed route simply returns a map of key value pairs. If you look at the route type (JsonRoute) you'll notice it's not a typical Spark route type. In fact it's a class implemented by me. JsonRoute, we'll look at the code shortly, makes use of a Spark class - ResponseTransformerRoute which is a specialized route type that supports the ability to "transform" the returned response model (the object that get returned by the routes handle method). In this implementation, below, I'm simply passing the model into a Jackson ObjectMapper,

public abstract class JsonRoute extends ResponseTransformerRoute {

    private final ObjectMapper mapper = new ObjectMapper();

    protected JsonRoute(String path) {
       super(path, "application/json");
    }

    @Override
    public String render(Object model) {
        try {
            return mapper.writeValueAsString(model);
        } catch (Throwable e) {
            throw new RuntimeException(e);
        }
    }
 }

This allows us to abstract out the process of transforming API responses into JSON and serving the correct response type as well. Yep I could have done this directly in my route declaration but I like to keep the main entry point for my application clean and readable as that's where most people will start to reason about how the application behaves. So that gives us a nice API endpoint. We can access this by running the application and hitting http://locahost:4567/api.

Serving Static Assets

Next up we want to be able to support serving the JavaScript, CSS and HTML that makes up our non-api aspects of our solution. Spark makes this easy

staticFileLocation("/public");

This command (declared in the main method) will wire up a handler to serve resources that are stored in src/main/resources/public as static files. You can also specify external files (outside the built jar for example) but I've went with resources in this example. Then we create an index.html page in this folder along with some JavaScript and CSS,

<!doctype html>
<html>
    <head>
        <title>Template</title>
        <link rel="stylesheet" href="/css/styles.css"/>
    </head>
    <body>
        <h1 id="css-broken-warning">The stylesheet didn't load or something, maybe</h1>
        <h1>Template</h1>
        <script type="text/javascript" src="/js/app.js"></script>
    </body>
</html>

The CSS file in this case hides the CSS Broken Banner to prove this works

#css-broken-warning {
    display: none;
}

And the JavaScript simply appends a new H2 node to the document, again just to prove the whole thing works.

(function(document, undefined) {
    var element = document.createElement("h2");
    element.innerHTML = "JavaScript Works";
    document.querySelector("body").appendChild(element);
}(document))

You'll rip these dummy files out when you actually being to write your app and it's better to write test for this sort of thing generally.

We can run the app again and browse to http://localhost:4567/index.html and make sure everything works.

Serving the Root

If you go to the root URL right now (http://localhost:4567/) you see that we still haven't satisfied the first condition of our SPA structure, we need to navigate to the index.html page specifically to see the app in action. To support the route URL serving the index.html file we can use a Spark before filter.

before(new Filter("/") {
    @Override
    public void handle(Request request, Response response) {
        try (InputStream stream = getClass().getResourceAsStream("/public/index.html")) {
            halt(200, IOUtils.toString(stream));
        } catch (IOException e) {
            // if the resource doesn't exist we just carry on.
        }
    }
});

The filter, locked down to the / route checks for the index.html resource and spits its contents out via a halt call (stop processing filters and handlers for this route). If it doesn't find it it just passed responsibility to the remaining filters and handlers. But why a filter? Well you could use a route and do something similar,

get(new Route("/") {
    @Override
    public Object handle(Request request, Response response) {
        try (InputStream stream = getClass().getResourceAsStream("/public/index.html")) {
            return IOUtils.toString(stream);
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }
});

But I think the filter is a bit neater. I the route case we have to return something. We can't just pass on to the next handler so instead we throw which may not make sense in all cases. Again it comes down to need here and YMMV so you now have 2 approaches to do the same thing. So running the app and hitting http://localhost:4567/ gives use our SPA infrastructure in place.

CORS

Supporting CORS in any web framework is a matter of Header manipulation - if you can do that you're sorted. Spark is no exception. CORS can be configured in a few different ways, you can set a few different types of headers, specify different header values etc. The example I provide here aims to give you a bit of configurability but uses it in the most permissive way possible (allowing any origins, header values and HTTP methods).

private static void enableCORS(final String origin, final String methods, final String headers) {
    before(new Filter() {
        @Override
        public void handle(Request request, Response response) {
            response.header("Access-Control-Allow-Origin", origin);
            response.header("Access-Control-Request-Method", methods);
            response.header("Access-Control-Allow-Headers", headers);
        }
    });
}

This method, which should be called early and called once in the main method, attaches a Spark filter to every route (you should alter that to your needs) that sets the necessary CORS headers to the passed in values. We can call this from the main method like this,

enableCORS("*", "*", "*");

Which is as open as possible (more than it needs to be in most cases) for remote clients. You can test this using test-cors.org on your local instance.

Now we have a simple CORS enabled setup to build you SPAs using the wealth of technologies that Java offers.

^{1: You could, if you desired, return HTML snippets, or JS commands rather than data but for the purposes of this post lets assume an API as this a lot nicer when supporting 3rd party apps that want to integrate - DATA IS KING and all that.}

Adventures in Clojure Land

Sat Jan 18 00:00:00 UTC 2014

Over the last few evenings I've built a very simple application using an exclusivley Clojure stack. This is my attempt to reason about that experience. It's not supposed to be a tutorial/how-to and should be taken as "here a some things I looked at and found interesting/useful, look at them too". If you want to see the resultant code you can get it via the depression-test repo.

I've dabbled in Clojure a few times in the past but nothing even slightly serious and usually going no further than some REPL tinkering so most of this was still very new to me. In fact my previous SSD died a horrible death so I even started on a fresh machine. I decided to approach the experience by intentionally over-engineering the solution, I included libraries that I really didn't need for such a small scale solution. I did this because there is an associated cost of entry for each new library you use and the increased surface area I was exposing myself to meant I'd feel pain that basic "Hello World" examples don't really give you.

Tech Stack

In terms of the tech stack I ended up using I had,

Leiningen - the build to of the Clojure world. Also included a few plugins,
lein-ring - start/stop your ring based apps
lein-cljsbuild - compile your ClojureScript into JavaScript
Ring - Web server/middelware library (like connect in the node.js world)
Compojure - Routing library that works well with Ring
Hiccup - DSL for building HTML pages/snippets in Clojure
Liberator - library for building RESTful web apps.
ClojureScript - I decided to build the front end interactions with ClojureScript and a few libraries,
react.js - shiny new JS framework for building UIs
cloact - a clojurescript-ified library on top of react.js
cljs-ajax - a neater ajax library

Leiningen & Plugins

Lein appears to be the defacto build tool for Clojure projects. Yes, you can use Maven or Gradle or whatever but there are certain benefits to using Lein.

Its written with Clojure projects in mind so fits better
The build file project.clj is just a Clojure data structure and incredibly easy to grok when in the Clojure mindset
It feels, in my limited exposure, really simple. Dependency resolution, adding new plugins, using custom templates or generators to structure new apps - all of it "just works". Admitedly I didn't do an awful lot with it but having done plenty of work with SBT and Maven it was incredibly simple but no less powerful to use.

lein-cljsbuild feel essential when working with ClojureScript. It seems to be the only tools that transpiles ClojureScript in JavaScript.

lein-ring is nice for getting up and running quickly but having just took a stab at deploying my app to Heroku it appears to be a bit troublesome down the line (though most likely down to my inexperience).

Ring, Compojure & Hiccup

I'd not be suprised to find that this trio underpinned most Clojure based web apps. Seems like a brilliant combination that works really well together. While my app doesn't use ring directly it's no less essential and as your app grows you'll invariably find yourself adding ring middleware directly to customise your app. Compojure is excellent for succintly expressing the routes in your app.

(defroutes app-routes
  (GET "/" [] views/index)
  (ANY "/questions" [] resources/questions)
  (route/resources "/")
  (route/not-found "Not Found"))

In this small sample alone I'm using the defroutes macro, GET & ANY verbs, the route/resources handler for serving static assets and the route/not-found handler for dealing with 404's in a custom way. Even with my novice mind this was incredibly easy to construct and easy to read. These routes are then turned into just another bit of ring middleware and added to the stack,

(def app
  (handler/site app-routes))

The app here referes back to the configuration in my project.clj that the lein-ring plugin uses to mount our application in the web server

:ring { :handler depression-test.core/app }

The handler/site part simply wraps our routes in more ring middleware that are common to web sites (e.g. middleware for extracting HTTP post form parameters etc.)

Finally hiccup gives use a Cloure DSL for building HTML blocks and pages. The index view is built with it,

(def index (html5
             [:head
              [:title "depression-test"]]
             [:body
              [:div#stage]
              (include-js "/app/app.js")]))

While this page is essentially static an culd easily be represented as an embedded resource you can see that this sort of expressive DSL allows us to dynamically construct views because, as is incredibly common in Clojure, it's just data.

Liberator

The inclusion of Liberator here was overkill but it was extracted from a slightly larger project I started at the same time as this. Liberator sits on to top of ring and focuses on allowing you to build RESTful APIs using a ridiculously easy to reason about structure. Usually when I work with frameworks or libraries that attempt to model RESTful APIs in a generic way theyend up being incredibly complex. Liberator is quite the opposite,

(defresource questions
  ;; return all the people in the application in json format
  :available-media-types ["application/json"]
  :handle-ok (fn [_] data/questions))

defresource is a macro used to generate a resource that can be used in a route. In fact my (ANY "/questions" [] resources/questions) route is basically taking any request to the /questions URI and letting liberator deal with it. Our resource definition simply specifies that this endpoint can deal with JSON media types and will coerce our questions list into a JSON response. It does this thanks to a bunch of defaults and an incredibly predicatable decision tree. If you want to accept different verbs, handle authentication etc. the approach to doing it is straight forward. Liberator have went to the effort of mapping the decision required to the turn our above resource into a JSON reponse via the mind boggilingly awesome decision graph which will show you what hooks and behaviours you can work with to make the magic happen in your RESTful API. I feel Liberator is one of those important technologies. I've built a lot of APIs in recent times and the sheer "obviousness" of Liberator is the polar opposite of the unpredictable magic in many other frameworks.

React.js

Everyone is talking about react.js - the new hotness from Facebook. I'll not dwell to much on it. I recommend to you check out this video around the design decisions of react and how its avoiding the usual terribleness of two-way data binding and state management. While my resultant app doesn't use react directly I did originally write the UI in Javascript/React before re-writing it in ClojureScript/cloact.

Cloact

One of many ClojureScript wrappers around React. This one appeared the most straight forward. It uses it's own atom implementation to manage state and change propogation

(ns depression-test.core
  (:require [cloact.core :as cloact :refer [atom]]))

(def selections (atom [1 2 3 4]))
...
(swap! selections (fn [_] [9 7 6 3]))

By declaring selections as a cloact atom then later using the atom api (swap!) to change its value any cloact component that uses the selections atom will get re-rendered throught the usual react method. The render-component method of cloact kicks everything into action,

(cloact/render-component
  [:div.questions
    [:h1 "Could you be depressed?"]
    [:p "This test will help you to assess whether you could be suffering from depression."]
    [:p "Answer the questions based on how you've been feeling during the last two weeks."]
    [score]]
  (.-body js/document))))

cljs-ajax

Nice little shim over the normal ClojureScript ajax stuff. Not much else to say,

(defn load-questions [callback]
  (GET "/questions" {:response-format :json
                     :keywords? true
                     :handler #(callback %)}))

IDE/Editor

I ended up writing most of the code in Sublime Text 2. I've never quite got my Vim setup right on this new machine and I've no clue about Emacs and ST2 did the job. I installed a few plugins to make working with a LISP a bit nicer.

SublimeREPL

Interactive REPL for Sublime that supports, among many other things, Clojure and ClojureScript. The REPL is where you end up doing most of the work in Clojure eventually pasting the same code into a file when you are done. SublimeREPL supports leiningen so you have access to your dependencies and app code when working in the REPL.

Paredit

Paredit mode comes from Emacs and makes working with LISPs parenthesis a bit easier by making sure you can unbalance your braces.

lispindent

Provides more lispified logic for dealing with indenting.

All of these are installed via the Sublime Text package manager.

Thoughts & Comments

Clojure

Clojure is a LISP and immediatley feels different to every other language I commonly use, but regardless, when you actually work with it there is a certain feeling of "naturalness" about the whole thing. The homoiconic nature, where "everything is just data" means that solving problems is suprisingly intuative and this applies to using libraries and frameworks. Most things are just lists and maps and you can inspect these things easily in the REPL. The amount of experimental spikes that work more or less first time is incredibly high and the final code is usually very terse. I'm by no means an expert, not even "average" at Clojure but it's a joy to use thanks to its well thought out foundation.

Another thing I've noticed is that Clojure libs dont suffer from the same API volatility that you see in likes of Node and Ruby. For example some popular libraries haven't had commits in 7 months - and it's not because they have stagnated or been neglected - it's because they're done. IMHO thats the sign of an incredibly mature, design concious community.

That said - some documentation leaves a lot to be desired if it even exists. I did often find myself searching to try and find what I wanted to know and usually resulted in just reading the code.

ClojureScript

I enjoyed working with ClojureScript but I didn't quite get my workflow right. ClojureScript takes a while to compile but this can be automated. Problem was I couldn't get this automation to work with lein-ring and so I found my self waiting ~20 seconds to try each change - it's cool though, I write Scala regularly. That said ClojureScript intrigues me more than any of the other JS transpilers around - I very quickly stopped seeing the point of CoffeeScript for example. The fact it works well with node.js (ClojureScript all the way down) is equally interesting.

Onward

Clojure, which was/is, very new to me helped me put together a simple application rather quickly as to how it stands up against larger projects remains to be seen. I'm going to try and see if I can improve my ClojureScript workflow as its a rather compelling technology but right now its a bit slow to use (e.g. I got better results using meis and lein cljsbuild auto but that skipped the server stuff). I also need to re-work this app to work on heroku without sacrificing the ease of lein ring server.

Its worth caveating that the code should in no way be assumed to be best practise, in fact its probably awful.

Convenient Colouring of Console Output in Scala

Mon Jan 06 00:00:00 UTC 2014

If you're writing console based applications or anything that may write meaningful information out to a console/terminal then you should probably consider making the output less bland. It's not just because people using your software like shiny things but it helps identify the severity or context of certain output. Look at tools like grunt or any of the modern test frameworks compared to maven. For a start they are much less noisy, maven is a bit of a chatty cathy, but they also make clever use of colours - red for bad things, green for good etc. This is something that maven could do with, discerning useful information out of the monochrome wall of text that maven spews out is a slow task.

In a recent Scala project this was something that I needed to do and found the default way of doing it rather noisy. For example writing out a red line and resetting the output requires this,

print("${Console.RED}[ERROR] ")
println("${Console.WHITE}$errorMsg")

Thats just to spit out a tiny bit of red, imagine what this looks like for more complex examples. So to tidy this up a bit I created an implicit class to add some colour features to strings (I come from a C# background and always liked sensible use of extension methods, and therefore implicit classes).

implicit class ConsoleColorise(val str: String) extends AnyVal {
  import Console._

  def black     = s"$BLACK$str"
  def red       = s"$RED$str"
  def green     = s"$GREEN$str"
  def yellow    = s"$YELLOW$str"
  def blue      = s"$BLUE$str"
  def magenta   = s"$MAGENTA$str"
  def cyan      = s"$CYAN$str"
  def white     = s"$WHITE$str"

  def blackBg   = s"$BLACK_B$str"
  def redBg     = s"$RED_B$str"
  def greenBg   = s"$GREEN_B$str"
  def yellowBg  = s"$YELLOW_B$str"
  def blueBg    = s"$BLUE_B$str"
  def magentaBg = s"$MAGENTA_B$str"
  def cyanBg    = s"$CYAN_B$str"
  def whiteBg   = s"$WHITE_B$str"
}

You can use this class in your code by importing it and then,

print("[ERROR] ".red)
println(errorMsg.white)

You can see it in action below,

Continuous Delivery & Boy Scouting Your Deployment Pipeline

Sun Jan 05 00:00:00 UTC 2014

Talking to teams and stakeholders within organisations over the last few years I've seen very strong concerns that continuous delivery was not for them. Usually it was because there was already a process in place - a process that involved lead times in the weeks or months, a lot of repeated manual work and many phases of QA prior to any release being deployed by some third party vendor. They worry, especially, that they would have very little chance of automating everything in the process.

Throwing the baby out with the bath water

The goal of continuous delivery isn't actually about automating everything (it just so happens that in many cases automation has significant benefits). Continuous delivery is about reducing the time to deliver a single change to production so to establish a tighter feedback loop between your company or product and your users.

However a lot of the time taken to deploy a release may well be out of your control, usually for the reasons quoted above. The thing is though, things change. It takes time but things always change. To quote Dr. Ian Malcolm - "Life finds a way". For example - your QA department might well suddenly discover the joys of automating a lot of their manual regression tests, your service management contract may expire and some new executive might suggest you try this new fangled "agile cloud as a service stuff" on the back of promises like 10x productivity ¹. There are lots of reasons, and most of them a blessing in disguise even if they are a bit ill-informed. Being ready allows you to capitalise on this organistational change and make it last.

Visualise your Pipeline

Draw a diagram or outline the steps that a change or single commit goes through to get onto a production box. This isn't done enough in projects, there are fragments of it in peoples heads and there are foggy areas of uncertainty. Drawing a pipeline, one that covers multiple deparments is actually quite an eye opener.

Another thing that this task does is help you realise that deployment isn't a single big black box that is always as slow as the slowest step. Its a series of steps and by visualising it you can simulate changes to it. That 1/2 day it takes to deploy to a preview environment for developer sanity checking may pale in comparison to the 4 week period that you wait to see if the production deployment was successful but bringing that 1/2 day down to 10 minutes for a team of, say, 6 developers has a more long term benefit. Even more, if you get to extend the process for deploying to preview into the QA environments at a later date - the work is mostly already done ².

So you may not have hugely reduced the time to a production release but you've helped tighten up the development feedback and prepared yourself for a future where these changes can be reused further down the pipeline.

Boy Scouting

But a lot of this work is far from trivial and right now it might be hard to sell it to the people holding the purse strings and Gantt charts so what can you do? Well, a few years ago Robert C. Martin (Uncle Bob) suggested that you can apply the Boy Scout rule to code bases.

The Boy Scouts have a rule: "Always leave the campground cleaner than you found it." If you find a mess on the ground, you clean it up regardless of who might have made the mess. You intentionally improve the environment for the next group of campers.

Applying this rule to a code base means that every check in you should,

tidy up a little bit of extra code,
add another useful unit test,
refactor a tiny bit of code, or,
tidy up a bit of formatting

Anything to have a positive impact on the quality of code in the project.

I proffer that the same rule can be applied to your deployment pipeline. Things like - introduce a script that starts up a simple vagrant environment, add a little bit of Puppet/Chef/Salt to create a production-like environment on developers machines, automate that file copy/paste process that Steve in Team B does during every release with a script, document the deployment process, publish the deployment pipeline, add a bit more validation to your CI server. Anything that makes the pipeline that tiny bit less painful to flow through.

In time these small incremental changes have a compound positive effect that results in fewer issues with deployment (and on-boarding) and acts as an example to people involved downstream in the deployment pipeline.

Ultimately the technical change required to move towards continuous delivery takes time and effort but it is nothing compared to the cultural and organisational changes. Proactively addressing the technical changes that can be made now can benefit your current process and also prepare you for a more effective delivery model in the future.

^{1: Both of these things have happened in recent years to projects I've been involved in.}

^{2: We haven't even pointed out that this developer deployment is going to be of immediate benefit for on-boarding new team members as well.}

Building Systems: Libraries and Frameworks

Sat Jan 04 00:00:00 UTC 2014

My Micro Service Architecture post has been by far my most popular post and has generated a lot of discussion. A recurring theme across these discussions is that of using small "fit for purpose" libraries or micro frameworks over larger frameworks. So I wanted to capture some of my thoughts on this.

What is Small/Large?

My view, as already expressed in the original post, is that smaller frameworks and libraries should be favoured over larger frameworks when building micro services. In fact thats my general rule of thumb when building systems of any scale. But of course this is entirely subjective - what I define as small is probably considerably large in some other peoples views (especially as we are being completely language and platform agnostic here). But there are a few things I consider when deciding if a library/framework is small or large,

The goal of the library/framework. Small things set out to fix a single problem, big things aim to be a panacea. jBcrypt does strong password hashing, Apache Commons does everything. ANORM gives you some simple sugar over JDBC, Hibernate has its own flipping query language for crying out loud.
Prefer libraries over frameworks. The difference being you call libraries whereas frameworks call you - frameworks are Soviet Russia. This definition makes libraries inherently more composable than frameworks. Libraries are often less prone to abstraction distraction (or at the very least their abstractions are rather specific to their goal). Proper usage of libraries allows you to roll your own framework and conventions rather than workaround generic conventions baked into frameworks.
Feature count. Kitchen-sink libraries or frameworks are often just full of features you don't need. "Well don't use them then" - ah if it were that simple. The unix philosophy should apply to services and libraries (do one thing and do it well). I mean, Apache Commons? C'mon Really? Why not just use a language with a decent core library instead?
Useful/Redundant Feature Ratio. If your framework/library has more features I don't need than ones I do, or if I need to use a feature I don't need or want just to use the other features - thats big. This is one reason why I consider Spring to be "big". It's entire premise is built around a big IoC container. The way I build systems I simply don't need an IoC container ¹. So being forced to use something I don't need to gain other features I can get elsewhere without the overhead - well thats big ²
Tooling support. If your library or framework needs a special IDE or admin application or other 3rd party installs - its big. Not necessarily "bad" but "big".

Why Small?

There are a million analogies you can use here, like "I don't buy a taco kit just to get taco shells" but lets try and stick to the reality here. I'm also going to avoid the obvious stuff like "frameworks lead to leaky abstractions", "frameworks lead to compromise or extra work" - there are plenty of articles about that stuff and YMMV anyway.

Firstly there is the obvious point that if you running a large number of micro services on a small number of machines then footprint matters, extra stuff loaded into a services memory allocation that aren't used creates a lot of waste but this doesn't really apply to most small/medium sized systems in reality.

Secondly design. Large frameworks are designed with rather monolithic architectures in mind. Modular service based systems have significantly different design needs that are orthogonal to the feature sets of large frameworks. This leads to questions like "Why not just use Spring for this instead of Dropwizard, I know Spring and you can do the same thing with them?". Ultimately it leads to design decisions that are more suited to monolithic architectures resulting a sort of schizophrenic architecture that becomes difficult to reason about. You'll have to actively work against other peoples design decisions instead of establishing your own.

Finally there is an argument that frameworks reduce the LOC metric which I think is

a terrible metric in the first place, and,
just plain wrong.

Just because you're taking other peoples work, neatly tucked away in a jar or package, and writing some of your own code around it doesn't mean you've reduced the LOC metric in any meaningful way. Yeah you've written less but if thats all you care about... mon dieu!

In fact the minute you take on other peoples code you've become responsible for it, you need to have some understanding about it, you need to maintain it (security holes etc.) and one day you'll stepping through that code to understand why things behave the way they do. Working with libraries in this context is actually a lot easier than frameworks, they typically have a much smaller surface area. When it comes to upgrades as well libraries can be significantly easier to manage without affecting other aspects of the system. The quick win you get with frameworks when you start a project will need to be repaid and you better be capable to explaining how your system works - the customers don't care that its a known bug in Rails - it's a bug in the system you delivered to them, you need to fix it. Starting with nothing and knowingly justifying every library inclusion you make will make the process of delivering and maintaining systems much more fluid instead of the promise a huge productivity spike at the start ³.

Oh, and don't get me started on the fact that many developers see unused features as a challenge.

I've had much better long term success and predictability by avoiding large frameworks and instead composing solutions using only what we needed even if it requires a bit of extra work up front or convincing. Admittedly a frameworks promise of productivity is a difficult thing to move people away from, especially if they married to the framework or don't really think beyond their own role.

^{1: IoC containers are suited to special types of systems - ones that have very large, deep or complex dependency graphs. In other words poorly designed systems.}

^{2: I know of a project that uses Spring inside their Dropwizard services. It's like people don't realise that you can actually call constructors using new. Not to mention the fact the initialize() method in a Dropwizard service is essentially an IoC container. If you're putting a container inside a container you probably need help.}

^{3: This sort of thinking is articulated really well by Dan Bodart in his Crazy Fast Build Times talk. His main argument is the tightening up of feedback loops but there are plenty of points that apply here too.}

So Long, and Thanks for All the Code

Thu Jan 02 00:00:00 UTC 2014

2014 is going to be a bit different. At the tail end of 2013 I decided to hand in and work out my notice with Kainos. So from February onwards I'll be doing a spot of contracting, validating a few business ideas and generally seeing how I can benefit the tech community in and around Belfast as a free agent.

Recently the cost of being a full time employee in a company that is growing rapidly has outweighed the value I've drawn from it and committing to another year wouldn't have been the right decision for me. So, instead, I'll be giving up the warm comforting embrace of full time employment with a company that has given me plenty of support through the years to rough it in the wilds of the tech industry. Hopefully this move will offer me greater flexibility and control in what I do.

As of now it seems I'm going to be pretty busy for the first part of the year but if there is anything commercial or even community related you feel I could help with (this site should tell you everything you need to know) don't hesitate to contact me.

LocalOnly Actions in Play!

Tue Dec 17 00:00:00 UTC 2013

A LocalOnly action in Play! is essentially an action on a controller that can only be called from the machine this it resides on. This is useful for restricting some endpoints (perhaps ops focused or database seeding tasks) from being called by all and sundry.

def LocalOnlyAction(f: Request[AnyContent] => Result): Action[AnyContent] = {
  Action { request =>
    request.remoteAddress match {
      case "127.0.0.1" | "::0" => f(request)
      case _ => NotFound
    }
  }
}

You can use this Action wrapper like so,

def index = LocalOnlyAction { request =>
  Ok("Local Only Action Executed")
}

This can be useful in the early stages of a project when your design is constantly influx and you need to perform actions on running systems. As your system grows this is perhaps a less favourable appraoch and you should consider moving these sort of tasks out of the application entriely.

Activate on Heroku

Wed Dec 11 00:00:00 UTC 2013

Activate is an object persistence framework for Scala. Unlike an ORM that openly tries to map DB concepts to objects Activate goes up a further level making the persistence aspect almost completely transparent. You work with objects and in doing so persistence happens as a side effect. For example given a class that extends the Entity trait,

class Person(var name: String, var age: Int)

We can create and persist this instance like so

transactional {
  new Person("James", 33)
}

Updating it will result in the persisted state of the instance being mutated as well. Activate uses the concurrency model of Software Transactional Memory at its core. As there is often some discrepencies between the conceptual model the database domain and objects Activate uses Dialects that handle the translation between the two (for example persisting an object with a List of Ints in a relational database that doesn't support array columns). In fact persistence is abstracted suitably that it becomes completely pluggable - use a document store like Mongo or a relational store like Postgres. Activate attempts to make this irrelevant.

I want to caveat all of this with the fact I'm not an Activate expert. I've succesfully used it in small volume projects and it's allowed me to be pretty productive. Which leads me to onto using it with Heroku.

Adapting Activate for Heroku

I have used Activate on 2 projects hosted on Heroku recently and I wanted to share a snippet that I wrote for the first one that I've found useful in the second. When you use Activate you need to create a Context that configures Activate for your application domain. One thing you need to set is the storage mechanism. On Heroku I used the provided Postgres instance which is made available to your application via the DATABASE_URL environment variable. So in JDBC based applications you need to coerce this into a JDBC connection string. To do this in Activate I used the code below which may come in useful for people wanting to do the same thing.

object PersistenceContext extends ActivateContext with Config {
  val storage = new PooledJdbcRelationalStorage {
        val uri        = new URI(System.getenv("DATABASE_URL"))
        val jdbcDriver = "org.postgresql.Driver"
        val user       = uri.getUserInfo.split(":").head
        val password   = uri.getUserInfo.split(":").last
        val url        = s"jdbc:postgresql://${uri.getHost()}:${uri.getPort()}${uri.getPath()}?ssl=true&sslfactory=org.postgresql.ssl.NonValidatingFactory"
        val dialect    = postgresqlDialect
      }
}

Scala for C# Developers

Thu Dec 05 00:00:00 UTC 2013

sbt-tasks - Rails/Rake custom tasks for SBT

Wed Nov 27 00:00:00 UTC 2013

Tinsmith: Online Scala REPL/Worksheet

Sun Nov 17 00:00:00 UTC 2013

I recently released a basic self-hosted, we based Scala REPL called Tinsmith (alternatively play with the online demo). If you've been following Scala news recently you may have noticed a similar project codebrew.io being talked about and in fact Tinsmith is heavily built off the same stack.

So why bother doing yet another REPL/Worksheet when one already exists? Well the first spike came into existence because codebrew.io was rather sluggish one day I was using it and I wanted something I could run locally or deploy to a dedicated remote instance when I needed. Then as I had my own instance running there were things I realised I wanted to do differently or didn't need at all and so rather than cloning codebrew I decided to use some common techs to produce a slightly different experience.

Tinsmith doesn't execute code automatically instead you need to explicitly execute code by Alt-Enter or ⌘-Enter.
Tinsmith doesn't have the bells and whistles such as Doc lookup and autocomplete
Tinsmith doesn't have Github integration (though it did have gist integration at one point but I decided against it)
Tinsmith does save the code to the querystring to make pages bookmarkable (like jsconsole)
Tinsmith does return the console output as well as the REPL output
Tinsmith does explicitly include some useful libs (Scala Utils, Scalaz, Guava etc.)
Tinsmith does run as a single application and can be quickly deployed to Heroku
Tinsmith does use WebSockets instead of Ajax

This is in no way meant to act as competition to the clearly better codebrew.io - it's simply something I needed quickly and decided other people might benefit from it as well. Afterall it's just the scala-codesheet and codemirror thrown together with a bit of webby glue.

tinsmith - Self Hosted Scala REPL

Wed Nov 13 00:00:00 UTC 2013

Play! 2 for Java Pluralsight Course

Fri Nov 08 00:00:00 UTC 2013

A few days ago my Pluralsight course - Play! 2 for Java was published through - you guessed it - Pluralsight. Its aimed at people new to the framework and is intended to act as a primer for building a simple app.

So if you're a Pluralsight subscriber or you've snagged one of the free passes that are readily available at conferences and meetups you should certainly checkout my course - a good response will invariably lead to me producing more videos so it's win win, right? But does this mean my old video series is done and dusted? Well no, the approach and delivery between my short information blasts vs Pluralsights more detailed approach are miles apart and honestly doing the course has actually piqued my interest in doing more screencasts soon.

This also explains my blog silence over the last while lets just say weekly travel with 4AM starts and late finishes, 2 kids and a video production does not leave a person much free time. Combine all this with the fact that video editing and mixing is a skill I've yet to develop it all adds up to a rather stressful time. That said Pluralsight are actually incredibly understanding and patient and if you've got something to share I recommend to contact them.

Hope you enjoy it.

Web Summit

Fri Nov 08 00:00:00 UTC 2013

Last week, along with 9999 other people, I attended the Web Summit in Dublin. The Summit is touted as being Europes answer to SXSW and honestly it's certainly not my usual type of conference but, hey, when someone offers you a free media pass you don't turn it down. And so this post exists because it's the least I could do.

So was it the European answer to SXSW? Pretty damn close in both spirit and execution. It was only 2 days which is much shorter than the SXSW marathon but still by the end of the second day I was nearly burnt out.

I spent most of my time in the Developer Stage/Tent - a stage curated by Eamon Leonard of Engine Yard and a few other Engine Yarders. Key takeaways for me included -

(Micro) Service Architecture is here in force. Essentially taking the Unix command philosophy of "Do one thing and do it well" and apply it to service oriented architectures. This leads to resilient self healing systems. In fact Chad Fowler made the excellent analogy of systems being like the human body where cells (services) are constantly dying and being replaced yet the whole system continues to run.
Immutability doesn't just apply to code. At Wunderkit/6wunderkinder they have an excellent story around immutable infrastructure. They practise it so much that their approach to spinning up new machines is to "throw away the SSH keys" so that the only thing you can do is create or destroy machines.
Testing is overrated - super contentious but the general arguments seem to be around typical testing techniques such as unit and integration testing. This idea fits well with MSA because if it is extremely quick to spin up new services or multiple version of the same service then business and operational metrics with automatic rollback to a "last known good version" trumps any form of extensive testing. I'm not saying DON'T TEST but you need to remember unit/integration tests for a volatile service are just extra work. When the service becomes more stable, maybe then think about how you wrap it in a safety net of extra unit tests? I dunno... we could write books on that debate but they've echoed many of the questions I've had myself.

The rest of the conference I floated around, seen many startups, shook MANY MANY hands. The evening events where nothing short of epic. Dame Lane was closed off and Summiters could wander between bars getting bathed in food, drink and music.

As I said at the start this wasn't my typical event as evidenced by the fact I spent most of it geeking out over technical things rather than taking in the Digital Marketing & Cloud stages. It's cool they had something for everyone in a city that I missed (which made it a welcome visit).

So great thanks and applaud go out Paddy Cosgrave and co. for a well put together event.

biomatic - Twitter Bio Generator for Realz!

Thu Nov 07 00:00:00 UTC 2013

Dropwizard - Production Ready Web Services

Wed Nov 06 00:00:00 UTC 2013

Play! 2 for Java (Pluralsight)

Fri Nov 01 00:00:00 UTC 2013

Better Management Through Agile Teams

Thu Oct 17 23:00:00 UTC 2013

Wait is waste! Value stream mapping, as Wikipedia reads, is a lean manufacturing technique used to analyse and design the flow of materials and information required to bring a product or service to a consumer.

The process of visualising a value stream is valuable for many reasons, one of which is that it can highlight bottlenecks or wait states. A wait state is a period of time in which actors in the value stream are waiting for something to happen, e.g. waiting for another dependent actor to finish their task. Actors in a wait state are typically not being utilised and therefore wasting time and money. Visualising the stream and quantifying the cost of these wait periods can often lead to fundamental organisational change.

In plan based or waterfall delivery wait states are ever present, at least from a single project perspective. All phases have a certain lead time while actors (analysts, developers, ops, testers etc.) start work and feed back more work to the other actors. The larger the phases, the longer that lead time is (within limits of course). Developers develop while testers wait, Testers test while developers wait. So how do you deal with these wait states? Typically speaking a customer isn't going to be to fond of paying for people to twiddle their thumbs. Depending on the period of time the wait state is expected to last you're either going to have to take the hit or repurpose the actors elsewhere. Repurposing is a very common practise - you'll see analysts swoop in at the start of a project write up a big document and slingshot onto the next project, resourcing has be carefully orchestrated so that a developer can roll off one project onto another that is about to start development and of course this all requires that the project goes according to plan - its like a giant game of Jenga. In the middle of a riot.

These practises may help reduce visible wait states but fail totally at solving the root problem - you've simply diverted peoples attention. You've traded off the measurable for the unmeasuarable. You've lost valuable insight by losing your analysts, increased the time taken to deliver features to due ramp up time for new developers and reduced effectiveness of testing because all the testers usually have to go on are out of date documents. Its hard to make a change when you don't realise doing it badly. The other glaring issue here is that the practitioners aren't the ones covering the cost of these problems - its the customer.

So what of agile & lean? The agile/lean model actually tends toward reducing wait states naturally. The model promotes 2 things

an "everybody, all at once, from early on" (Lean Architecture) mentality. This is realised in the form of a team being the primary unit of delivery as opposed to a specialist with the support of a team.
Delivery in small batch sizes. Deliver in smaller increments reduces lead time between deliverables. In fact any large constraints that can potentially create lag become immediately apparent and need to be dealt with to facilitate quick return.

The team analyses, the team develops, the team tests, the team deploys, the team doesn't need to wait. Start to finish the people that know the most deliver the solution and they do so by delivering small chunks of features to the customer to support continuous and evolving understanding of the solution. This reduces any potential bottlenecks from knowledge loss or misinterpretation of what the end user wants.

If you're coming from an organisation that is used to cycling siloed roles and specialists (analysts, developers, testers) the transition to a more team focused cross functional model isn't an easy one but the result is a more effective, predictable approach to delivery which benefits both yourself and the customer.

On Evangelism...

Sun Jul 28 23:00:00 UTC 2013

So i'm a Developer Evangelist now.

Officially.

For Kainos

Now it might seem a bit odd being a developer evangelist for a company that doesn't have product or service but I strongly believe it's a vital role regardless of company type. The fact that Kainos now has a Developer Evangelist is a public declaration that we have a duty of care to help grow the technical communities we affect. It's not to say we didn't do this before but now it's out in the open - we can start setting goals and measuring our successes and failures.

I want Kainos to be recognised as a great place to work for people wanting to grow their technical career - not through random word of mouth or shiny recruitment campaigns but by real honest community engagement and commitment to contributing to Open Source. Initiatives that were once internal by default will be external unless there is good reason (customer confidentiality being a key one). Doing things in the open not only helps other people but provides us with a fresh perspective on things we've probably taken for granted. We have so much technical expertise across a huge range of technologies and we want to discover ways of sharing the things we do in a more public forum.

This is not about sales, marketing or recruitment - all these things are happy side effects if I've done my job correctly. No this is about showing we genuinely give a damn. This is about building trust in the developer communities around us. This is about finding the people in our company that want to speak at conferences or work on open source projects and giving them that support. This is about people and community. This is about craft. I know it's cheesy but I truly believe in the concepts of Software Craftsmanship - it's about genuinely caring about the work you do.

Of course those that know me will know none of this new to me, hence why I'm the first of many developer evangelists that Kainos will likely empower but the fact I have time and budget to do this stuff means I can ramp up my activities and experiment. Try Things is one such experiment and I hope by the year is out to have started something interesting and engaging with it in both Belfast and London.

This is only the beginning, here goes nothing...

^{* It's probably worth nothing that while the role is full time this doesn't mean I immediately distance myself from day to day project work - that would be pointless. It is simply not possible to extol the virtues of "craft" without practising it. I'm going to remain delivering projects and livin' the dream.}

Try Things

Tue Jul 23 23:00:00 UTC 2013

I've always been a tinkerer. I value understanding different things. Even if the net result is "eugh that's a terrible idea". I enjoy challenging my assumptions and opinions because being proven wrong means I've learnt something and that's good. And while I may never use the language or tech I've tinkered with there may well be principles or practises I can apply to whatever I'm doing presently. Generalism is in my blood. But tinkering can be a lonely old game, finding like minded people wanting to tinker with the same things with the same availability is a thankless task.

It's time to try something new - it's time to Try Things. Try Things is a hands-on meet up format aimed at the technically curious.

How It Works

Try Things is a relatively informal concept but the general flow of an event is outlined below. It is a technically focused event so a computer is somewhat essential for attendees.

Pre Event

First things first you'll need a backlog of topics/suggestions. Write an app, use Trello, index cards, suggestion box, email, it doesn't matter really. Topics can be anything really,

An emerging language like Elixir or Clojure
A new database technology like RethinkDB or Datomic
A new build tool like SBT
An architectural concept like Single Page Applications
An idea like mobile time tracker

Kicking off the Event

Duration is entirely open but a good 3-4 hour window provides a perfect buffer, though if you are a particularly effective driver then I'm sure 2 hours is great.

Once people are gathered and settled one of the previously submitted topics are selected at random. If you want to turn the randomness down a tad you can also use a bit of consensus voting. First randomly picked topic to get a majority thumbs up is your topic for the event.

Challenge Setting

Depending on the topic there may be goals, challenges or questions that the participants may have. Perhaps different people want to look at different aspects of the topic, maybe they want to approach the topic in a different way or perhaps they simply thirst for competition. Before the main part of the event kicks off people should quickly share what they know about the topic, raise any interesting questions and highlight any learning goals they want to get out of it. These should be recorded in some manner to allow the participants to reflect back after they've finished hacking.

The Hack

Once people are happy and if necessary divided into teams everyone gets to the hacking. Its important to keep the communication open at this point as you don't want people going back into their caves during this time. This is supposed to be a social thing remember. A few things to promote the social side,

Have a vocal facilitator doing the rounds and highlighting interesting discoveries
Scheduled coffee breaks to allow people to lift their head above the screen and flex their vocal chords
Team rotation

Wrap Up

Once the hacking has come to some satisfactory conclusion or designated deadline it's time to share what we've learnt. This can be simple show and tells or checking our progress against the initial objectives/challenges.

The end result isn't supposed to be a black and white "yeah I'll use this thing again", it's supposed to be a shared learning experience and provide an avenue for people to work on things they may not get a chance to in their daily work.

The Next Step

I'm intend to run a few Try Things sessions over the next few months in London and Belfast (dates and locations still TBD) so if this seems like your sort of thing and would be interested in attending, or if you have any suggestions (or maybe you just like the potential of free beer) - please let me know on Twitter (@kouphax)

So Unbecoming

Wed Jul 17 23:00:00 UTC 2013

The post has been updated to accomodate some of the excellent adivce from Roland Kuhn via his Gist

I've been playing around with Akka lately for a component I've been working on. The component, a simple remote file sychroniser will essentially batch download files from URLs returned by a remote API call. The specifics aren't important right now but thats the gist. The component is scheduled to run every 3 hours (using the Akka scheduler) but sometimes the entire download process can actually take longer than 3 hours and I don't want to end up thrashing the remote API for little benefit - so I wanted a fail safe to ensure a new download process would only start if the previous one had finished.

So I've got Akka and some kind of co-ordination requirement. There were 3 possible options,

Make use of a global var
Use Akkas FSM (Finite State Machine), or,
Use an Actors become & unbecome methods

We'll start with some akka system boilerplate (with scheduler) and look at the options

// create the overarching actor system that will manage our application
val system = akka.actor.ActorSystem("devtracker")

// import our execution context
import system.dispatcher

// set up a scheduler to sync the registry
system.scheduler.schedule(0 millis, 3 hours) {
  // try and start a download
}

Along with this we'll have an actor that performs the download and notifies something when it is finished.

The Global `var`

Simplest appraoch would be to just use a flag,

// create the overarching actor system that will manage our application
val system = akka.actor.ActorSystem("devtracker")

// import our execution context
import system.dispatcher

@volatile var isDownloading = false

// set up a scheduler to sync the registry
system.scheduler.schedule(0 millis, 3 hours) {
  if(!isDownloading) {
    system.actorOf(Props[Downloader]) ! Download
  }
}

Then somewhere in our Downloader actor we'd set global isDownloading flag to true then, once the (synchronous) download process is complete, back to false again.

class Downloader extends Actor {
  def receive = {
    case Download => {
      Application.isDownloading = true
      // perform download
      Application.isDownloading = false
    }
  }
}

Ugh mutable state, am I rite?!?. It's no fun having to manage shared mutable state (even if it is currently just only boolean flag). This is especially when you're using a system that aims to abstract away the concerns of concurrency.

Lets not forget that I've also broken the whole encapsulation of the actor model by explicity accessing Application and that just seems dirty. Of course you could keep refining this unitl you ended up with an actor that isolated the var and other actors passed messages via the Actor System (or its Event Bus) but, well, thats already kind of been done for us in Akka.

The Akka FSM (Finite State Machine)

The rules we described above are pretty much describing a very basic state machine. The downloading component is either in an idle or downloading state and we can only initiate a download if the component is idling.

Akka provides a basic FSM implementation which is essentially an actor with a neat DSL for defining states and transitions. We can model our download case like so,

class DownloadCoordinator extends Actor with FSM[State, Unit] {

  startWith(Idle, Unit)

  when(Idle) {
    case Event(Go, _) => goto(Downloading)
  }
 
  when(Downloading) {
    case Event(Finish, _) => goto(Idle)
  }
 
  onTransition {
    case Idle -> Downloading => {
      context.actorOf(Props[Downloader]) ! Go
    }
  }
 
  initialize()
}

The code above tells us that we have 2 states Idle and Downloading. When we receive a Go event (a simple case object) in the Idle state we move into the Downloading state. When we receive a Finish event in the Downloading state we go back to Idle.

Finally, on the transition of Idle to Downloading we tell our downloader actor to do its thing.

When the download completes our actor can simply tell it sender it is finished

class Downloader extends Actor { 
  def receive = {
    case Go => {
      // perform download
      sender ! Finish
    }
  }
}

We don't need to directly tell the sender but here this is acceptable. The level of decoupling and indirection is entirely up to you.

Finally our scheduler simply asks the co-ordinator to Go

val coordinator = system.actorOf(Props[DownloadCoordinator])
system.scheduler.schedule(0 millis, 2 hours) {
  coordinator ! Go
}

Now if the scheduler ticks while a download is active it just gets ignored (though you can optionally handle it in the whenUnhandled block of the FSM actor)

I'll come back this implementation near the end of the post.

Become/Unbecome

Do you find the FSM implmentation a bit wordy for something that only really has two states? There is a lot of extra stuff going compared to the boolean flag appraoch. Well actually actors are capable of being their own FSM without the need of the FSM trait.

Akkas Actors support a pattern of swapping out the actors message handler for another receiver via the become/unbecome methods. We can implement our download co-ordinator with alot les code like so.

class Downloader extends Actor {
 
  import context._

  def receive = {
    case _ =>
      become {
        case _ => // do nothing
      }
  }
 
  def download = future {
    // perform download
    unbecome
  }
}

And thats it. This entire thing replaces both the co-ordinator actor and downloader actor. Our scheduler remains roughtly the same except we call this actor instead of our co-ordinator.

So whats happening? Well when we receive any message (for the first time for example) we call download which begins the download process. Then our actors becomes something else - the thing it becomes is a cold, uncaring machine - doing nothing to any message it gets. When the download completes the unbecome puts the actor back into its initial (idle) state.

Conclusion

So what did I personally end up using? I went with the FSM approach. Why? Well I think, even with the basic nature of our needs, the FSM approach is actually more understandable. Comparing my two solutions (which may or may not be good practise, I'm new to Akka) I find the become/unbecome approach to be rather cryptic - sure there are less lines to take in but thats a crappy metric to measure quality by. That said reaching for a fully fledged FSM strategy for every situation is probably going to grow out of control - so it may not always be the best case. Know thine weapons.

Scala - For Fun & Profit

Fri Jun 28 00:00:00 UTC 2013

Elsewhere

Wed Jun 05 23:00:00 UTC 2013

Oh hey hi, this post is a "personal" post. It doesn't involve any code or analogies to writing software (that I'm aware of) so yeah.....

Many many years ago (well about 9 or 10) I went through a period of "being a singer/song-writer". There was no intention of making money or even playing live - I can barely speak to people I don't know let alone "perform" in front of them (though I can waffle about technical stuff Ad nauseam - oh the irony) - I just felt compelled to record some noise.

Experiment #1

The first time I did it was for my sister, a born and bred muso, as a birthday present - she had moved to London a few years ago and I wrote a song about her being always "busy".

The quality is clearly terrible, the song is clearly terrible, I am clearly terrible. But dear readers - hello - this is me. Was me.

Anyway. I quite liked the process of recording - one maybe, two takes, no editing. I hated hearing my voice and shambling guitar but I did like the fact something existed that I created.

Experiment #2

After that I did a little experiment. The idea was this

Drink for an hour
Record an hour of music (one take)
Drink for an hour
Record an hour of music (one take)
5 and 6 and 7 and 8 follow a similar pattern.

That was awesome because it was, for me, kind of wild and cool and if I ever put any thought into anything I hate it.

Experiment #3

Also out of the previous process I got enough rough recording to do an Extended Play of music I genuinely like to this day.

This was a cathartic post - I did things, and yeah I do things.

Culture Bubbles Up

Sat Jun 01 23:00:00 UTC 2013

As an organisation grows the way by which culture is formed and played out changes. In companies with smaller numbers - culture is more organic, typically people are like-minded and share a similar goal. As a company grows and diversifies (different products, projects or locations) culture becomes more broken down or siloed across the projects or locations. Typically speaking companies of this size share a common structure - top-down whereby key strategic decisions come from the top down.

Culture is different, culture comes from the people that interact with the company the most - the lower levels of the org chart were head count is greater and interaction is more inward. Project teams typically form their own unique cultures but these aren't reflected at a more broader company level and are often short lived as people move around the company.

But culture, at an organisation level, is a key differentiator for a company. Culture defines the type of people that work in your organisation. This isn't news to anyone. Companies know this, but it's how these companies deal with this determines what culture is actually formed.

For IT companies, whose core is that of technical folk, technical culture plays an important role. The right technical culture typically leads to the right technical people. Sure the company BBQs and Christmas Dinners are all well and good but motivated technical people want to learn and share beyond what they get on a project.

In my travels I've come to group companies into three distinct bucket categories when it comes to how they support or react to technical culture,

Prohbitive

Some companies attempt to stimulate technical culture the same way they run their business - by pushing mandates from on high. Forcing culture is doomed from the offset. For example - I know of a large IT company that wanted to hire more developers. The powers that be had heard that developers like hackathons so they decreed that the company shall run an internal hack event to promote their engineering division. It was a marketing ploy and like all marketing work it was handled by marketing. Marketing picked the problem to solve , marketing picked the participants (Solution & Technical Architects that represented the brand) and wrote the story before the event took place. That was the last anyone heard of it - I'm not even sure if the event even happened. To most people that hear this story and hear the name of the company it's of little surprise.

The point is you can't create culture that defies how you allow people to work.

Permissive

Better companies are more permissive, they provide financial support and/or time for internal initiatives. This is much better than trying to force anything but it still has its problems. I've already mentioned that culture flows in a different direction to strategic company decisions so being permissive is only part way there. An employee asking for support to run an event can feel like going against the grain and if it happens at all still needs to be justified exactly like all business decisions (whats the return on investment, how do we measure the success etc.). Problem is culture is so fuzzy and unpredictable its hard to come up with an answer that isn't just nonsense. This all creates friction and cultural change is hard enough without extra impediments.

Proactive

The best IT companies are not only permissive but senior level people are proactive and open about supporting a technical culture. Rather than the management of culture that happens in permissive companies, great companies lead culture. They instil a "go do" attitude in people rather than "ask first". They do this by being proactive in letting people know what they can do, they see culture forming and actively push that up, they allow failure. Rather than seeking funding for something, funds are made available up front. If someone decided to run an impromptu hack session on a new technology after work and expensed a couple of pizzas there would be no questions asked. I'm a firm believer in the old adage of "It's better to seek forgiveness than to ask for permission" and great companies usually are too. When you have a culture that the people directly affected by that culture can control, you'll have happy people. Not only that but you'll have the right type of happy people.

Personal Experience

My current employer, who are actively growing, recently published some material to highlight their internal culture and the thing that struck me about this material was that two of our culture differentiators as a company was hackathons and CodeSlice (a monthly out-of-hours tech meet up) and I'm rather proud to admit that both of those initiatives came from myself and a bunch of other ne'er-do-wellers (Will Hamill, Jonny Holmes & Luke McNeice). All it took was a few short emails to our CTO and we were in business. That was 2 years ago and since then various parties have been championing CodeSlice on a near monthly basis (which spawned similar in other locations), ran 4 internal hacks (which also spawned others in other locations), been key organisers at 2 external hack events and had teams attend a number of other hack events (notably the NHS Hack Days). You can't buy that sort of enthusiasm but you can nurture it and let it grow.

Disciplined Agile Delivery

Mon May 27 23:00:00 UTC 2013

Thanks to Liberty I had the opportunity to attend a morning session by Scott Ambler around Disciplined Agile Delivery.

The selling point around DAD seems to be that it offers a more complete view of both project and organisational delivery than what Scrum does. The argument being that Scrum gives you a framework within which you can execute project delivery through sprints but doesn't itself address what happens before (where did the product backlog come from) or after (how are we actually releasing the product to production). So Scrum teams start adopting other practises to roll their own, more complete, process. Of course this sort of approach should really be done by "Process Experts" and as most people are not "Process Experts" the resulting process is typically sub-optimised. DAD suggests that rather than starting at the baseline and optimising to the middle, why not just start at the middle?

This is somewhat analogous to the old libraries vs frameworks debate in the technical world. Frameworks make decisions for you and as such are great when venturing into a new technology space but as you become an "expert" in that area the helpfulness of frameworks diminishes and they can become rather restrictive. At this point "experts" begin to craft their own solutions typically utilising a small number of libraries. I like to consider my self well versed in development of solutions and as such I typically avoid big frameworks and favour smaller, more lean bespoke development.

In this sense DAD is a framework, it gives you everything, and in comparison Scrum could be considered a library, giving you a small set of tools and as such I'm inclined to tend toward rolling my own process utilising various methodologies and processes. This has worked for my projects.

But here's the rub - While I may be a "technical expert" (tending toward the use of libraries over frameworks) I am by no means a "Process Expert". My current experiences have been with smaller scale projects and a bespoke process will fit well at that scale without much expert knowledge. We do, however, need to move beyond that; toward proper organisational change and without "Process Experts" the approach of mashing some principles and methodologies together will result in a sub-optimised process tending toward failure.

The process aspect of large scale software delivery is my "new technology space" and as such it may be prudent to investigate a large focused framework until such times as we can adapt and move beyond its limitations. Will it be DAD? Will it be the Scaled Agile Framework? Perhaps it will be like the carefully governed but organically grown Spotify style model. I don't know yet, but I do know its time to start moving toward using agile in the large and I doubt a slapdash approach will be enough until I developed a broader understanding into larger organisational challenges.

trello-printer - Print cards from Trello boards

Fri May 24 00:00:00 UTC 2013

The Old "Agile is for Developers" Myth

Tue May 07 23:00:00 UTC 2013

There is a common misconception among people who are just starting out on the agile path that the agile model favors developers and downplays other roles. While it is true that agile puts a greater focus and added responsibility on developers it also broadens the definition of developer from the niche of "people who write code" role to a more umbrella term of "people who deliver software".

Another comment I heard being made, admittedly in jest, is that agile is "Disneyland for Developers" and honestly this couldn't be further from the truth. In fact agile methods typically expect people involved in delivery to be MORE responsible for delivery. Developers are no longer just mindless code monkeys banging out tasks against pseudo-code written in a document. No, developers are expected to DEVELOP a working solution. This includes analysis, design, architecture, development & testing. All of this is expected to be done with one eye on the "big picture", they are expected to understand WHY they are doing something from, both, a technical and business perspective, they are expected to be at least understand how their code is deployed and often expected to be able to deploy it. You don't have the Java guys, the Web guys or the DBAs - no, a single person in a team should at least be capable of driving out a vertical slice of functionality from infrastructure to database to services to HTML and JavaScript. Of course the slice doesn't have to be, nor will be, perfect right now. There will always be space for specialists to come in and make their areas better, more secure, faster, whatever needs done. But at the very least when all but one person on the team catches some rare exotic disease on a team bonding night out that person can still deliver valuable working software until the rest of the team recovers.

Ultimately though the agile methods that are technically focused are about ensuring quality of delivery. Delivered features should be "done" rather than thrown together an handed over a test team to let them find the issues - a common side effect of time constrained, fixed scope, phased delivery. That practice is unacceptable even in plan based or phased models and regardless of the project approach a development team can and should be making use of practices often associated with agile.

But other roles necessary in the delivery of a project also benefit greatly from agile approaches. Just think of some of the things these other roles would do within more traditional project delivery models,

Who has to spend days or even weeks manually running the same few hundred scripted tests every few months to avoid regressions and assert correctness of the system? Not the developers thats for sure. Testers are often required to execute manual scripted tests which are slow, prone to error and have a relatively low ROI compared to other testing techniques. Manual tests check for issues you know will break your system, but its the unknown issues that typically kill your solution. Automation of these tests (by developers and testers) gives faster more frequent feedback and allows testers to actually do interesting, rewarding, more valuable testing such as Exploratory Testing. Testers should not be there to break things but to ensure we build the right thing right in the first place.
Who spends weeks going through multiple versions of documents that seldom get updated and go out of date shortly after sign off? Again, not typically the developers. Business Analysts and Solution Architects spend a long time producing documents that create a contract for all parties involved. Often, though, they are only pulled out during "disputes" around agreed functionality (between either developers & tester or customers & analysts). If that happens though, then who is actually reading these things properly? No matter how rigorous you make documents there will always be the problem of interpretation. You can't misinterpret a running solution.
Who is expected to "manage expectations" when delivery is slipping beyond the scheduled dates?. Long running projects that pin all their plans on up front estimates will slip. It's inevitable unless you've managed to overestimate everything but then I'd wonder why you are doing the work and not your more aggressive competitor. When this happens it's typically the job of the manager and senior technical folks to inform the customer in the best way possible, to "manage their expectations" (shudder). No one wants to do that. In typical agile models visibility is a key differentiator (another way to ensure quick and frequent feedback) which removes the need to for big bang disappointment. When things change or slip you know about them much quicker and can course adjust, the impact is significantly reduced.

All these (arguably wasteful) practices are typically the responsibility of people in non-developer roles. Now that we have reduced or removed these practices does that mean we've made the roles redundant? Not at all - as with the developer role increasing in scope so does the manager, analyst and tester. Testers no longer just "test", they have a more important function - assuring that what they test is the right thing, they've become crucial to analysis. Analysts also need to understand aspects of project management and managers themselves can move to a more hands-off sales and account management role as delivery is managed by the team itself. What we will see is a reduction of specific titles on projects, most people no longer fit those niche titles.

Facilitators & Implementors

So, yes, agile approaches certainly favour working software over volumes of documentation describing what the system should do but heck - so does common sense.

Agile approaches flatten the role structure across the delivery of a project. You end up with umbrella terms like facilitators (who ensure the right thing is being built) and implementors (who ensure the thing is being built right).

You go from this,

To something more like this,

If you wanted to map these intentionally nebulous terms back to typical roles in a more traditional model you have,

Facilitators - Business Analysts, Solution Architects, Customers, Stakeholders, Users, Testers, Project Managers, Legal etc.
Implementors - Developers, Testers, DBAs, Designers

Its clear to see where the commonplace agile roles fall as well - Product Owner and Scrum Master would be facilitators and the team would be the implementors.

Already, even at this high level you're getting overlaps and in many ways everyone plays their part as both a facilitator and an implementor but its generally obvious what a persons primary function is and you can plan around that accordingly.

You don't need fancy, lofty titles because you have a single purpose. Titles only force attribution of responsibility to particular people, they enforce specialism and defensive attitudes. Titles shouldn't affect the delivery of a solution.

So, no, "agile" is not just for developers, it's for the facilitators, the stakeholder & the implementors, it's an all-encompassing first step on the road to refocusing our goals from following a process to delivering the right thing quickly and painlessly.

webbit-metrics - A Metrics based Admin Server for Webbit

Thu May 02 00:00:00 UTC 2013

Micro Service Architecture

Mon Apr 29 00:00:00 UTC 2013

Micro Service Architecture is an architectural concept that aims to decouple a solution by decomposing functionality into discrete services. Think of it as applying many of the principles of SOLID at an architectural level, instead of classes you've got services.

Conceptually speaking MSA is not particularly difficult to grasp but in practice it does raise many questions. How do these services communicate? What about latency between services? How do you test the services? How do you detect and respond to failure? How do you manage deployments when you have a bunch of interdependencies? So lets expand on some of these throughout this post and see if MSA really is worth the effort.

Anatomy of a Micro Service

First things first what actually is a micro service? Well there really isn't a hard and fast definition but from conversations with various people there seems to be a consensus that a micro service is a simple application that sits around the 10-100 LOC mark. Now I realise that line count is an atrocious way to compare implementations so take what you will from that. But they are small, micro even. This means you're not going to find hundreds of tiny services built on top of large frameworks, it's simply not practical. No, simplicity and lightweightyness ^{(not a real word)} is the order of the day here. Small frameworks like Sinatra, Webbit, Finagle & Connect do just enough to allow you to wrap your actual code in a thin communication layer.

In terms of a footprint these services will be small, you're potentially going to run a lot of them on the same machine so you don't want to be holding on to memory or resources that you aren't intending to use. Once again simple libraries over large frameworks will win out, you'll also find less of a reliance on 3rd party dependencies.

This decoupling at a service level also offers another interesting option. We've pushed a lot of the old application complexity down to infrastructure level. We are no longer bound to a single stack or language. We can play to the strengths of any stack or language now. It's entirely possible to have a system built out with a myriad of languages and libraries, though as we will touch on later this is a double edged sword.

You're also not going to find any true micro service based architectures that are hosted in an application server, that kinds of defeats the point. To this end micro services self host, they grab a port and listen. This means you'll lose any benefits your typical enterprise application server may bring and your service will need to provide some of the more essential ones (instrumentation, monitoring etc.).

Communication

This is an interesting one. How do your services communicate? There really isn't a single answer for this, even in a single solution. The most basic approach across the board would be to expose all services over HTTP and pass JSON back and forth. Service discovery (how one service knows where to find another) can be as simple as putting the endpoint details into a configuration file (or simply hardcoded).

You may discover, in certain circumstances, the cost of serialising and deserialising JSON payloads through an entire transaction is causing bottlenecks. Perhaps JSON isn't suitable at all (binary formats) and so you'll want to look at different protocols, like Protobuf

But hardcoded urls kind of imply a certain strict coupling and A-B-C workflow. In a layered application this makes sense but moving into a service based architecture means you're not bound by this (potential) constraint. Some communications between services can be completely decoupled, in fact some services could publish events or data blindly. They can just throw it into the ether (or at least a message bus) and maybe one day some service can come along and start listening. Alternatively maybe parts of your system could operate in a batch/offline process, feeding off a queue and flipping bits many hours later.

A micro service approach gives you this kind of flexibility without major architectural rework.

Monitoring & Metrics

Components of a monolithic, layered solution don't typically fail silently - it either fails to compile or throws exceptions when something is wrong (unless you go out of your way and silently swallow exceptions like some insane person). In a service based approach a service might drop off the face of the earth and it's very easy for no other service to notice (especially in a pub/sub model). This means it's imperative that services are properly monitored and orchestrated. In fact just knowing a service is alive is seldom good enough. Is it providing enough business benefit? Is it even being used anymore? Is it acting as a bottleneck for reliant transactions?

Monitoring is always important but more so in a service based architecture where failure is often less obvious. Good examples of this from the JVM world are Metrics and Ostrich these libraries not only allow for collection of metrics but provide a means of integrating and reporting that data into other services like Nagios or Ganglia.

Testing

There is nothing particularly special about testing services in a micro service based system but the point I want to raise here is that you've reduced the need to have a full suite of tests for every service. Because a service does one thing, and should do it well the risk of introducing a system breaking bug is significantly reduced thanks to the natural behavior of a service based system. Now writing tests for the sake of it, or just in case, holds little to no value but still require the same amount of care or attention during refactoring.

I'm not saying don't test but I am recommending you think before you use the old "No. of Tests" as a value of code quality.

In fact lets go one step further and change the way we test entirely. You've got monitoring in place, at any given time you can know what services are broken and recover. But also why not monitor key business metrics on each service and use them as indicators to know if we've broken something. This is commonly known as a production immune system. If conversions are dropping, or sales have halted then you can make it so your system rolls back to a known stable state and gives you nasty looks until you fix the problem. The same approach can be used for A/B testing and Dark Launching new features and services.

Micro Service Architecture actually makes a whole lot of sense when it comes to ensuring you have a stable system at any scale.

Applications of Micro Service Architecture

MSA has a number of interesting applications. Obviously green field solutions of a given size is an obvious use of this approach. Though I say "given size" because the cost incurred at an infrastructure level required to support micro services could outweigh the benefits and, for small systems, a simple web app would be totally acceptable.

Micro Services also have an interesting application in large legacy systems. Working with legacy code is risky at best. For systems that have been running for decades there is a chance there is limited knowledge in how the systems actually works internally. Working with this code can be like working with a house of cards and a simple mistake in one place can have adverse affects elsewhere. These systems are typically mission critical so mistakes can be costly. So just don't touch it. The micro service approach lets you do this. Rather than diving into the legacy code base, write a small service that does what you need (yes you may be duplicating legacy code) and proxy service calls through it (e.g. via nginx/apache etc or purely within code).

Now imagine doing this regularly as part of your maintenance and support program. You'll be turning off the old system bit by bit and perhaps eventually be able to turn it off completely.

Issues

Micro Service based systems aren't without their problems of course. For one there is a greater need to better understand non-development areas. In days of old before DevOps was a broadly misunderstood marketing term developers would deliver working systems over the wall to operations who would deploy it to production and everything would fail. Of course this was the ops team fault, don't they know anything? Jeez. It was entirely possible for a developer to know only code, running the entire application from their IDE without thinking about how this would work in real life. MSA forces the developers hand to actually be more responsible and actually get involved in scripting of deployments and understanding integration requirements etc.

Another issue that I touched on earlier is the fact, left unchecked you could have hundreds of small services written in hundreds of languages. Simply managing that sort of estate of languages and platforms with their different installations, package managers and requirements can stop a project dead in its tracks.

An anti-pattern has emerged from the MSA approach - often called the Nano-Service architecture. These are systems built on services that are so small that duplication of effort is prevalent and it becomes almost impossible to reason about any particular area of a system easily.

Macro Post for Micro Services

The irony of a large wall of text to introduce the concept of micro services is not lost on me but there is lots to consider. Micro Service based systems can produce scalable, fault tolerant systems on fairly low cost hardware. MSA promotes good practices of keeping things lightweight, emerging systems and business monitoring but as with everything should be approached with a rational mindset and solid understanding of what you need to achieve.

Rice

Sat Apr 20 23:00:00 UTC 2013

Rice. A swamp grass that we consume, awesome. But how do you prepare rice? You could,

Buy it in pre-prepared microwave packets
Buy a rice cooker
Make it yourself using some water and a suacepan

Theres probably more but that would detract from my carefully constructed metaphor (I'm not drunk, honest). So whats the point? Well lets think about the pros and cons of each.

Buying In

You can buy these little pouches of rice that you pop in the microwave for 3 minutes and bing you've got hot steamy rice. Sort of. It's never perfect. The work involved in taking a perishable food stuff and making it "less" perishable, packaging it and shipping it around the world kind of degrades the product a bit. You ultimately have to trade off quality for convenience. You can certainly add ingredients to the rice to rejuvenate it but its never quite same and lets be honest the packet size is always either too big or too small and you end up wasting some.

Using a Rice Cooker

Rice cookers typically cook rice really well but they are an investment, you may only use it occasionally, some of them are a pain to clean and you've always got to find space for it in your kitchen (if you had a fancy tool for every type of food you'd need a much bigger kitchen). Plus these things don't really scale very well. They typically make a certain size and if you want more than that you need to start doing it in batches or invest in another rice cooker.

Make it Yourself

Rice can be a pain in the ass to cook, it sounds simple. Too little water and it burns and sticks to the bottom of the pan, too much and it boils over leaving starchy water everywhere. You can under-cook it, or wore overcook it into a stogy mess. If you get distracted or try to do too many other things its easy for things to go bad and you'll be left with a mess that needs cleaning up. Then theres the problem of judging the right amount of rice to cook. Eventually though, after a spot of trial and error, cooking rice without fancy tools becomes rather easy and predictable.

A Carefully Constructed Metaphor

I did mention previously that this was a metaphor and I wasn't lying. I've been mulling over this analogy for a while (while I was cooking rice for real). In software you've got similar approaches to delivery,

You can buy an off-the-shelf solution that accomplishes most things
You can use a framework with all the bells and whistles
You can roll your own solution

The all have their own trade-offs that align well with the rice metaphor.

Off the shelf products give you a certain level of convenience provided you are willing to live with that you are often compromising user experience to satisfy the view of a "typical user" that the product must adhere to to appeal to wide user base. And yes you can typically add your own ingredients through extra development work but it's never quite right.

By using a large framework you've got accept a certain amount of upfront complexity for the future promise of making the delivery of solutions easier. Much like the rice cooker, frameworks will invariably bring with them a slew of features that you'll never use. However there will always be the need to manage and cope with that extra complexity and often the cost of this complexity (setting up boilerplate for a new feature, writing a bunch of configuration to allow the framework to do it's magic etc.) could be greater than the value a small feature provides. Features users want will be deferred because of decisions they did not have a say in.

Finally the idea of making your own rice, or using a minimal set of common tools to deliver the right amount features. Yes it may seem initially daunting, you'll probably get it wrong a few times (do too much or not enough) but eventually you'll build up enough knowledge and skill that you can see that reliance on heavy abstractions is often more of a burden than a help. While rolling your own may appear as unnecessary risk - taking this approach in small steps is significantly less risky than pinning all your hopes in a frameworks sales pitch or the promise of buy-and-run COTS products.

Building your own stuff has the added future benefit of letting you make more informed decisions. Clearly there is no one right answer for every situation - sometimes you may just benefit from firing up a ready rolled WordPress site, or, perhaps Middleman or Jekyll would fit in well with the publishing strategy of the project, but sometimes a simple Makefile with a bunch of sed calls will do the job.

Finally the simple act of actually trying build your own solutions will give you greater clarity when you need to answer questions around build vs buy or technology choices. Just because frameworks, tools and products allow you to be ignorant of many things it doesn't mean you should be.

Hardening Sprints: Just Say No?

Wed Apr 17 23:00:00 UTC 2013

I was recently asked by a PM in my company if "hardening sprints" are allowed in agile projects and I gave the very pragmatic answer of "you do what needs to be done". I left the conversation not feeling right though and I've been thinking about the question ever since. To be honest my gut instinct is that, with certain exceptions, these sort of sprints are a smell and their origins are firmly rooted in phased/waterfall delivery. I'll address the "with certain exceptions" caveat a bit later but lets first dive into the idea of hardening sprints.

What is a hardening sprint?

A hardening sprint is a timebox or sprint reserved at the end of a group of sprints (usually prior to release) to allow testing or hardening of the release. This could mean extra testing, different types of testing, refactoring, reviews etc. It brings with it the rather bizarre concept of "Done Done" - i.e. features that have been delivered in previous sprints may be "Done" but not "Done Done". So it may be coded and "kind-of" working and "kind-of" tested and people are "kind-of" happy with it and yeah its "kind-of" done but we'll wait until the end to know if we are really done.

So What?

It strikes me as "kind-of" odd that this sort of thing happens without people raising an eyebrow. For one thing there is no such thing as "Done Done" and if we don't admit it I fear that we'll see "Done Done Done" pushing its way into our process vocabulary.

The next thing that strikes me - there is absolutely no data available to predict how long this "hardening" process should take. If we take it at face value the term implies a single sprint. How can we be sure whatever unpredicatable stuff comes up during that sprint can actually be addressed and adequatley resolved in that sprint? Or, taking it to the other extreme (indefinite amount of sprints) how do we know when to stop - afterall nothing is ever perfect.

Undoing All The Good Work?

So we've spent X amount of sprints refining our approach to delivery, continually improving and learning and now we've thrown the project into this huge dark pit of uncertainty and "hardening". Surely I'm not alone in thinking this sounds somewhat wrong.

Another risk is that this kind of strucutre will create is a potential reduction in ongoing quality. It could be argued that hardening sprints, much like the old phased/waterfall approach to delivery, removes a certain amount of responsibility from the delivery team. Now there is less of a desire for developers to apply as much rigour to their code as they may have - afterall "the testers will find the bugs so why waste my time being thorough?". But if you are deferring your user or penetration testing a few months down the line there is always this notion that the team can just "throw it in and see what comes out in the wash".

"With Certain Exceptions"

Of course, like every rule, there may be exceptions. One obvious exception is external penetration testing. There can be considerable cost and time assoicated with getting a third party specialist in and its certainly not viable to do it every sprint. So defering a full pen testing cycle until near the release is an acceptable and often necessary exception. But that doesn't mean the team should throw all care about security out the window - the goal of the penetration testing should be to validate that there are no vulnerabilities not to discover them.

Next Time Gadget... Next Time

So are "hardening sprints" a good idea? I'm inclined, as negative as it may be, to start with "no" and take it from there, afterall its better to start with discovering why we can't do certain things inside of a sprint rather than assume we can't and carry on ignorant of the potential benefits.

Teams or People?

Sun Apr 07 23:00:00 UTC 2013

A team is much more than a group of people thrown together for the period of a project. This is quite common in a utilisation based business model where downtime is viewed, on the balance sheets at least, as money wasted. I'd call that myopic.

Putting aside other arguments for a moment (downtime is quite often good for allowing motivated people to learn and grow) lets think about Tuckman's Stages of Group Development.

Tuckmans model suggests that teams go through various stages of development, as denoted above. During these phases various things happen,

Forming - the team comes together shares their history and begins to develop an understanding of the problem to be solved.
Storming - after the gentle introduction people begin finding their place in the team. Dominant parties fight for control, less dominant sniff out their leaders and the power struggle kicks off.
Norming - the power struggle is over and people understand their place in the team. Less impediments occur through communication.
Performing - the team has gelled and is working effectively, everyone is in the role they need to be and everyone trusts each other.

There is a final stage - Adjourning where the team, as a collective whole, part ways - but I'll get to that later.

What does the y-axis in this pseudo-graph represent? I guess the obvious answer is team productivity but given the many factors that can affect team productivity/efficacy this isn't a realistic metric. No, the y-axis in this case reflects the teams focus on their goal of delivering the solution. In this situation it makes sense that a "team" become a staple unit of delivery vs the old resource rumble of "whoever is free". Short-lived teams of randomly selected individuals are always going to be operating at sub-optimal levels due to the very nature of team dynamics. Now here's the rub - more often than not this performance is nigh on impossible to quantify. In fact if "people as units of delivery" has been a thing for a while you'll find people making excuses for inefficiencies by front-loading of estimates. What happens then in sales and bids, how can a company who's estimates are derived due to sub-optimal team output remain competitive but keep their process the same? One dubious tactic would be for sales team gently massaging numbers and rates to make them more appealing. This makes the entire delivery process from sales to final delivery of a solution highly inefficient and pressured and we haven't even touched on the evils of potential overselling and micromanaging!

So - the team a a unit of delivery makes sense IMHO, but it's not without it's problems. Moving to a team-based model requires a significant jump in company culture, accounting, sales and, if you'll let me use the word, resourcing. There are potentially huge visible risks in long stretches of downtime for teams. Arguably though large visible, almost predictable risks, are significantly better than unquantifiable consistent loses right? At least you can mitigate them.

Adjourning

A team can't stay a team forever. People get hit buses or worse get bored and a long lived team that refuses to change is simply another silo of knowledge that methodologies like agile try and crack open.

But adjouring the team isn't about throwing the core team to the wind. Too much team churn and you end up back a square one. Too little team churn and you've got the silo problem. Team members should be rotated to allow knowledge to propagate (inwards and outwards from the original teams perspective). However this clearly brings its own problems (is a person who's constantly rotated really ever part of a team) and the balance needs to be struck over time.

Spotify is decent example for this sort of cross-team knowledge sharing and I'd recommend checking out Scaling Agile @ Spotify

The team as a unit of delivery is certainly more suited to product based companies or large, long running projects because it feels more natural. In a services based outfit or consultancy house being able to ship people about at will is easier but thats not to say its actually beneficial. While it brings its own unique challenges team-based resourcing can allow organisations and project to greater realise their true potential.

The 'New' Value Triangle

Sat Mar 30 00:00:00 UTC 2013

A projects delivery effectiveness can be affected by many things. There is, however, a baseline trade off on three core values and being aware of this trade off can help you make better decisions. The diagram above highlight these core values,

New Domain - this can be a new problem domain or new challenges within an understood domain. It may also be considered a new domain if a large proportion of team have limited or no experience in it. (e.g.coming into the health sector for the first time or possibly new legislation coming into effect).
New Technology - New technology can be either mandated (integration work) or just by virtue of the fact technology is always evolving. There are always new (and potentially) improved ways to solve common problems in solutions.
New People/Team - New people or teams occur when a large proportion of a team has never worked together or are changed between phases. Depending on the type of organisation you work within you may never work with the same person twice or for very short periods of time. This is common within utilisation focused service companies for example.

The "new" value triangle proposes that,

Given a new project you must only accept up to a maximum of two of the "new" trade offs.

Obviously the less the better but there will invariably be challenges outside of the project that will have an impact on these values. Having to cope with all three simultaneously will seriously impede successful delivery of a project and must be avoided at all costs.

Its worth noting that the term "new" when looking at each trade-off on its own merit isn't necessarily a negative thing. New blood on the team can inject fresh spirit and allow knowledge to propagate beyond the boundary of the team, new technology can speed up delivery of more robust solutions and a new domain can bring considerable business value. But at scale or combined these "new" values can become a hindrance on successful delivery.

Another consideration is that this is a general rule. You are not always going to be in a situation that mandates such trade-offs. Or perhaps you are in a situation that seemingly mandates all of them? It's possible, with careful thought, to slowly transition through all the "new" values over time but at any given point its always wise to keep tabs on what you are trading off to ensure you aren't holding back on the effectiveness of delivery.

So you've got to know when to trade off certain values and when you do what mediation patterns could be applied to prevent unnecessary waste

Standardise Technology (NPT/ND)

When a team (potentially new to each other) is kicking of a project in a new problem domain (to the team or the organisation at large) there is usually the greatest business risk and as such the team need to be ready for surprises as, and when, they happen.

One of easiest ways to achieve this is to reduce risk on the technology front. Standardising and using the most boring, non-shiny, well understood technologies removes any requirement for the team to learn new paradigms and principles while trying to wrap their collective minds around a new domain and potentially getting up to speed as an effective team.

Commoditise Solutions (NT/NPT)

From an organisational perspective when you find yourself consistently in a well known domain there is a risk you can begin producing extremely similar systems time and time again. This is isn't going to look terrible on the balance sheet but if you have a high team churn there is a lot time being wasted in getting people up to speed that could be avoided.

This is probably a very good opportunity to commoditise your solution, turn it into a product and sell it. That way you can establish a baseline consistency via a core team and established technology stack and focus even more on the adding true business value.

Iterate & Experiment (ND/NT)

Given an established team entering into new domains and technologies the simplest way to reduce risk is to iterate & experiment rapidly. If the organisation you work within delivers bespoke software its very likely that the work you do will be within a new domain or at least a new/different subdomain of a common domain you typically deliver into. When introducing new technology as well it's vital that teams work well and can begin delivering quickly and iteratively to understand both the new domain and any potentially how new technologies fit.

Having a largely consistent team as your unit of delivery allows us to focus on delivering the right tools to tackle the right problems which adds significantly more business value than the other trade-offs. Of course - changes are perfectly acceptable & encouraged as long as they aren't wiping out a large group of the core team in one fell swoop.

Conclusion

At any point in any project there will be trade-offs. Many of these trade-offs can be boiled down to something similar to this value triangle and knowing strategies for dealing with these situations allows the project to optimise for the most effective approach.

Uploading Files with Play! 2

Mon Feb 25 00:00:00 UTC 2013

Source

A quick look at how to achieve file uploads with Play! 2 including other form fields and uploading more than one file

play-reactivemongo

Mon Feb 18 00:00:00 UTC 2013

Source | Official Site

ReactiveMongo - reactive, asynchronous and non-blocking Scala driver for MongoDB.

In this episode we will retrofit the app from episode 003 to use Mongo, ReactiveMongo and the play-reactivemongo plugin.

ReactiveMongo

Mon Feb 11 00:00:00 UTC 2013

Source | Official Site

ReactiveMongo is an asynchronous, non-blocking MongoDB driver written in Scala. This screencast will move quickly through the basic concepts that define ReactiveMongo and then show examples of the various basic operations you'd expect from a MongoDB driver

Starter for 10: Play! 2 - Part Three

Mon Feb 04 00:00:00 UTC 2013

Source

Part three of the Introducing Play! 2 Series. Play! 2 is a Java/Scala framework based on a lightweight, stateless, web-friendly architecture for highly-scalable applications. This episode will introduce us to the concepts of

Play Forms for binding request data
View Helpers, and,
Form Validation

Starter for 10: Play! 2 - Part Two

Sun Jan 27 00:00:00 UTC 2013

SOURCE

Play! 2 is a Java/Scala framework based on a lightweight, stateless, web-friendly architecture for highly-scalable applications. In this episode we will look at

Creating Controllers & Actions
Creating read-only views
Routes and URL matching

Screencasts with a JVM flavour

Mon Jan 21 00:00:00 UTC 2013

Today sees the inaugural launch of a new project/passion of mine - yobriefcasts.tv. There is a launch post about it on the site as well as the first in a series of screencasts around Play! 2 (more being edited as you read this no doubt). Initially I never set out to focus on the world of the JVM but after building a list of potential topics and doing some research there is just so much there and so little audio/visual coverage of it that I think it deserves some attention.

As it stands right now there is an [optional] price associated with the productions - you can stream everything to your hearts content or you can pay a minimal sum to get a high quality download of the screencast - you can do both if you want. It took a while to come to an acceptable price point and structure because I have an end goal here - I want to feed any money earned back into the project to get better equipment and thusly produce better quality screencasts. Futhermore I'd eventually like to turn this into a (mostly) full time gig and move to a subscription model with frequent (weekly) releases - but thats a long way away yet - for now I want to see if I can build something people want and enjoy.

But hopefully the site can speak for itself - I've already got an extensive list of releases coming up over the next while so stay tuned, follow us on twitter as @yobriefcasts and enjoy the site.

Starter for 10: Play! 2 - Part One

Mon Jan 21 00:00:00 UTC 2013

Play! 2 is a Java/Scala framework based on a lightweight, stateless, web-friendly architecture for highly-scalable applications. In this episode we will look at

Introduction to Play! 2,
Downloading & Installing Play!
Using PVM (Play Version Manager) to Install Play!
A brief tour of the command line interface.

Trello Cards: Materialising Trello Boards

Fri Dec 28 00:00:00 UTC 2012

TL;DR I ported pivotal-cards to trello-cards

I recently did a spot of work with the awesome and brilliant people of the Government Digital Service and being big fans of getting stuff done they make use of whiteboards and index cards to visualise the flow of work on the various projects. Thats all well and good but it makes reporting to senior managers, stakeholders and remote people a bit difficult (and for the uber paranoid - what about auditing!!!). To that end this flow of work was also mirrored online - and in GDS's case the tool of choice was usually Pivotal Tracker. In order to bridge the gap between the online and "real" one of the smart chaps at GDS wrote pivotal-cards which lets you generate printable index cards from you pivotal board. Lovely stuff

Anyway - I spent some time today porting this little project to Trello. I have been using Trello more and more on recent projects but always miss the physicality of real index cards and I'm usually strapped for time to start writing them out myself.

So here it is - trello-cards, I've tried to keep it pretty much 1:1 with pivotal-cards but there are some TODOs still outstanding - the current card make up looks a bit like this.

So basically follow the instructions on the project page and hopefully it should all work out. If it doesn't - FORK IT 'N FIX IT or complain and I'll fix it for you :)

Some other points worth noting,

Currently tested on Chrome 25 (OSX)
Could do with some basic tests around it
Currently no markdown support for descriptions
Story points use the Trello Scrum Chrome Extension convention of (POINTS) at the start of the card name/title. If you use that plugin then good for you, if not you can still make use of the points convention
Tasks are derived from the first checklist on the card

Eg. Play 2.1 + Scala + Guice

Tue Dec 18 00:00:00 UTC 2012

TL;DR Here is some sample code of how to use Play 2.1s GlobalSettings#getControllerInstance feature with Google Guice.

I've had the chance to take Play 2.1 RC 1 out for a spin recently and in a bid to make my project a bit more testable I decided to go ahead and give dependency injection a bit of a spin again. While things like the cake pattern and general dynamic composition are alright I've never been happy with the code that I produce using them. So I decided to give Guice a blast as it seemed very much aligned with how I approached DI in the .NET world.

My main need was to create instances of controllers and given Plays appraoch in the past (singleton objects for controllers) 2.1 new GlobalSettings#getControllerInstance feature was what I needed.

First things first we need to add a project dependency to Guice and while we could just add the Guice dependencies directly I went with a library that offered slightly neater syntax in Scala - SSE-Guice. In short this means instead of writing this sort of noisy code

bind(classOf[Service]).to(classOf[ServiceImpl]).in(classOf[Singleton])

We get to write it like this,

bind[Service].to[ServiceImpl].in[Singleton]

So lets add the reference to the Build.scala file (assuming we just generated a our Play project) for sse-guice

val appDependencies = Seq(
  jdbc,
  anorm,

  // Add your project dependencies here,
  "com.tzavellas" % "sse-guice" % "0.7.0"
)

Next up we need to modify the Application controller a bit

It needs to take a dependency
It needs to bo be instantiable (e.g. not an object - this might be doable but the default approach in Guice seems to conflict with Plays approach to defining an instantiable controller)

So we want to go from this,

object Application extends Controller {
  def index = Action {
    Ok(views.html.index("Some Message"))
  }
}

To this,

class Application @Inject()(val messeger: Messeger) extends Controller {
  def index = Action {
    Ok(views.html.index(messeger.getMessage))
  }
}

A few things of note here

We use the @Inject() attribute here to mark this constructor as our default one (need by Guice to satisfy constructor injection)
Application is now a class. Small change (a Levenshtein Distance of 6 even!) big difference. Rather than having a singleton we now have something we can construct when we want (even in tests) and set it up really simply. Construction and lifecycle can be handled via our DI library easily.
We are passing in some sort of dependency - in this case it's a simple string barfer (Messeger is a trait) and the impl looks like this

trait Messeger {
  def getMessage : String
}

object MessegerImpl extends Messeger {
  def getMessage = "The Best Super Message"
}

Then we need to create our Guice module that will wire up our dependencies,

import com.tzavellas.sse.guice.ScalaModule

class SimpleModule extends ScalaModule{
  def configure() {
    bind[Messeger].toInstance(MessegerImpl)
  }
}

The ScalaModule declaration is a Scala-fied version of Guices AbstractModule with some helper and whatnot. Guice applies some common sense to wiring so we don't need to declare each controller type here etc.

Finally we need to start the DI mechanism and provide a way to instantiate controllers. We do this via the Play feature of GlobalSettings. In our app package we just put a Global object and set it up as needed

object Global extends GlobalSettings {

  private lazy val injector = Guice.createInjector(new SimpleModule)

  override def getControllerInstance[A](controllerClass: Class[A]) = {
    injector.getInstance(controllerClass)
  }
}

We use a lazy val to instantiate the injector with our SimpleModule and override the getControllerInstance method of our GlobalSettings class to use Guices injector.

Oh one last thing. We need to tell Play what controllers we want to use this feature (it's not an all or nothing thing which I guess is good). We do this by marking the route declaration with an @ in the routes file. So it becomes this (notice the @)

# Home page
GET     /     @controllers.Application.index

Running the app gives us exactly what we expect but why stop there? We need to test this stuff right? Cool. Testing is made easier - we can simply make use of the various helpers in Play like FakeApplication and FakeRequest,

class ApplicationSpec extends Specification {
  "Using FakeApplication" should {
    "get the default message" in {
      running(FakeApplication()) {
        val home = route(FakeRequest(GET, "/")).get
        contentAsString(home) must contain ("The Best Super Message")
      }
    }
  }
}

Or we can create a custom Global object with a FakeApplication

class ApplicationSpec extends Specification {

  object DummyMessenger extends Messeger {
    def getMessage = "Dummy Message"
  }

  "Using FakeApplication" should {

    object DummyGlobal extends play.api.GlobalSettings {
      override def getControllerInstance[A](c: Class[A]) = {
        new Application(DummyMessenger).asInstanceOf[A]
      }
    }

    "get the an injected message" in {
      running(FakeApplication(withGlobal = Some(DummyGlobal))) {
        val home = route(FakeRequest(GET, "/")).get
        contentAsString(home) must contain ("Dummy Message")
      }
    }
  }
}

And even lighter we can just test the controller as if it was just another class,

class ApplicationSpec extends Specification {

  object DummyMessenger extends Messeger {
    def getMessage = "Dummy Message"
  }

  "Using direct controller testing" should {
    "return the dummy message" in {
      val controller = new Application(DummyMessenger)
      val result = controller.index()(FakeRequest())
      contentAsString(result) must contain ("Dummy Message")
    }
  }
}

Hopefully this gives you a good starter for using the new Play 2.1 getContollerInstance feature along with Guice.

Neo4J Talk: Using Graphs to Analyse Public Spending on International Development

Wed Dec 12 00:00:00 UTC 2012

Better late than ever, right? So a few weeks ago I presented a bit of a case study on my use of Neo4J in a UK government project focused on increasing transparency around how the UK disburses foreign aid. This post is a collection of all the useful resources mentioned in the talk.

Plenty of acronyms were thrown about and here are the useful links to some of the key ones.

GDS (Government Digital Service)
DFID (Department for International Development)
IATI (International Aid Transparency Initiative) - Registry and Standard
OIPA (Open IATI Parser and API) - Open Aid Site and Source

MongoDB Basics

Thu Dec 06 00:00:00 UTC 2012

Neo4J: A Case Study

Wed Nov 28 00:00:00 UTC 2012

Vertical vs Horizontal Thinking

Sat Nov 17 00:00:00 UTC 2012

I'm not quite sure were this ramble is going to lead, it's been a while since I've felt like writing something that hasn't been clearly defined by my half conscious brain but lets take the journey anyway shall we?

This is going to be loosely related to agility, not necessarily the "agile method or process" but more simply about adapting to change and embracing transparency in a software project. More often than not this can be simply referred to as "common sense" and it's a great thing.

But I digress,

In a vertical world you deliver features by implementing vertical spikes through each layer of the system architecture. It doesn't have to be right, but it does need to be good enough for now. Sure it may not deliver the best user experience or maybe there are more HTTP requests or database calls than necessary but it's useable. Its shippable.

In the horizontal world layers of the architecture are tackled by (possibly pseudo) specialists and delivered en masse. This is very typical of the Big Design Upfront, Plan Based, Waterfall approach to delivery but its visible in a lot of projects in various guises and it's rather natural to many people.

Now, on the surface, getting specialists to deliver what they know best and then smash them altogether actually sounds great. You know your DB person is going to produce a wonderful hyper-normalised schema for you, your UI/UX person will deliver a gorgeous mockup of the site and your middle layer app people will deliver the best truly ReSTful service layer known to man. But then you spend an insane amount of time (or many many little periods of time) doing the actual integration and making this vision a reality. The testing, the regressions, the edge cases the performance hits and sacrifices through integrating perfect works of horizontal art. Best case you get it done eventually, worst case someone else gets it done by layering hack upon hack at integration points just to get it working.

Now imagine your DB person gets eaten by a badger, or your UI person decides they've had enough and goes to live in a free-love commune and drink the spiked kool-aid? I've overly dramatised the Bus Factor here but the point is still made. You've lost siloed knowledge forever and you need to spend time building it up again, and time and quality slip.

Finally - projects that deliver horizontally are, more often than not, projects that have limited customer/stakeholder visibility. Sometimes even less for end-users. So delivering a mockup of the UI and a DB schema that can be exported as a nice UML diagram is enough to satisfy people that work is on-going and going well. Even if it isn't. It's not a lie but it is a false impression, and it's an impression that doesn't work for long especially if you have shorter release cycles with ongoing customer interaction. A page of beautiful buttons that don't do anything will create an impression that you are nearly done and can easily generate disappointment at subsequent demos. Is it worth it?

Don't get me wrong it is very possible to successfully deliver software in this manner, on time and on budget but so many things must fall into place and all the stars must be aligned that it's a rather big risk, but a risk that people seem to ignore.

So I'm saying vertical is better and there are many reasons for that and I'll touch on some of them here but there is one very fundamental reason it's better. Working and thinking vertically allows you to create a shippable product early. It's not all done, far from it. It probably looks like ass as well but still it's usable enough to get the job done, right? Imagine using a new product that looked amazing but only had 3 working features and the rest of the buttons and knobs didn't do anything. Wow factor is for kids and sales men.

Why else is vertical thinking better?

For one everyone on the team works across all layers of the solution. You don't have huge silos of knowledge contained in a single brain of some crazy bus magnet. Sure you may involve specialists to ensure you're not doing it totally wrong but at least you'd be working WITH them and picking up parts of the vast knowledge they may have.

By delivering something "good enough for now" you actually reduce waste. True you may re-write a few things to get it right but you haven't spent hours crafting perfection only to tear it apart. Even better the first iteration may make users realise - "Actually you know what? That super whizzy button thing - we don't need it". Waterfall attempts to "manage change" because change in waterfall is scary expensive. Embrace it early and often and it much less costly.

Finally, the solution evolves rather than being dragged into existence from a 189 page document of business rules and textual assumptions. This allows you to experiment and try things, throw them away or discover a better way of doing things. You can adapt the solution at anytime to suit the changing nature of the environment it lives in.

Vertical thinking is a proven pattern in many other aspects of the software world. TDD for example. Rather than writing reams of code and testing it manually later, you create a solution bit by bit and adapt it fearlessly as you go. UI designers never just bang out one design and leave it there. Smaller feedback loops are essential to flow and success and vertical thinking gives you this. Break the habit of wowing people and replace it with the habit of delivering - it's a better, longer lasting feeling of win.

Technology Techsmology

Mon Nov 05 00:00:00 UTC 2012

There is an old adage,

Projects don't fail because of technology decisions

The younger me would have got all uppity and defensive when he heard this statement. As a developer/architect type whose main goal is to make key technology decisions on a project (not to mention as a person who likes to grandiously consider themselves a craftsman) this does seem rather belittling. Are you telling me all my decisions are fairly irrelevant? I'll just stay in bed then shall I?

But you know what? The statement is also completely true.

But hey, wait - do you want to know something else? Yes? Good.

The world of technology is not black and white, projects aren't either a success or a failure, it's simply not a binary system. Projects may not fail, but they could be late. They could be well over budget. They could have code bases so toxic that you need to hire contractors to do the work because no one wants to touch it. They could be so overly laden in patterns and complex architecture that even small changes take weeks or months to implement (and who foots the bill for that?).

Technology decisions can indirectly ruin the morale of your staff, muddy the relationship with your customers and generate really high turnover of staff in your company (which may be a good fit for some companies but you can never foster excellence if you don't nurture talent).

This statement can also be used as a weapon - a weapon to excuse bad decisions and hold back change (both good and bad, in short it stiffles learning). People cling to technologies that they used in the past, for fear of more pain with other technologies. They shoehorn their existing skillset into problem domains that aren't a good fit. When this happens, at the extreme you end up with specialists that simply can't adapt to shifts in technology fast enough.

So yeah, projects don't fail because of technology decisions. But you know what - if your dentist turned round to you with an evil grin and a rusty chisel and Maslow's Hammer citing - "No surgery ever went wrong because of a lack of anesthetic" - you're not likely to get in to that chair and open wide. And even if you did (maybe, you're drunk, maybe people keep telling you this dentist is the best, whatever) the experience would be painful, recovery time would be longer and subsequent treatments would probably be more painful. A rather shambling metaphor I admit but I write what my brain dictates - I have not choice in the matter.

Finally, I fully accept that many of these decisions are out of our control. Customers, especially big ones, generally have the final say, they've potentially invested a lot of time on a certain stack and have a lot of legacy (read: code that still works) solutions in play. True, but one of our goals is to at least inform these people of the alternatives and let them make an informed decision - maybe not today but soon enough the cost of developing and maintaining new systems built on out-of-support JDKs will come under scrutiny and they'll need to consider alternatives, alternatives that you've already enlightened them about.

Neo4J Talk in November

Fri Nov 02 00:00:00 UTC 2012

Its been rather quiet around here of late, truth be told I've been rather busy with my work (this peak is a rather large one). Being busy for me usually implies I'm not getting bored on cookie cutter, by the numbers web apps, being busy means I'm doing new interesting stuff. I've been doing a bit of work with the UK Government, specifically DFID (Department for International Developement) around the nobel goal of transaprency and open data. The project involves interrogating a lot of highly connected data in weird and wonderful ways which can mean only one thing - Graph Databases. We tried the relational database route early on and it was causing too much pain to get the data we wanted,

So having recently been enlightened by Jim Webbers talk on Neo4J I decided to try the Graph Database appraoch and it paid off, in spades. Then one humble brag on Twitter later and I get asked to present a case study on Neo4J at the London User Group - and I'm stoked. This is the first talk I've ever given that I don't have to pretend to know what I'm talking about - awesome.

If you're in London on the 28th of November come along and see how I've (probably incorrectly) used Neo4J.

CASE STUDY: using graphs to analyse public spending on international development
Wednesday, November 28, 2012
6:30 PM To 7:30 PM

Skills Matter
The Skills Matter eXchange
116-120 Goswell Road
London EC1V7DP

RSVP

Learning Resources for Scala, Play! and Mongo

Fri Oct 05 23:00:00 UTC 2012

I've been asked a lot recently for some resources on various technologies, namely Scala, Play! (2.0) and Mongo so I thought I'd share the wealth (so to speak).

Got any good resources that I haven't listed here? Let me know.

Scala

Official Site
Screencast
Various Links from the official site
Scalatron - learn Scala by programming a game
Scala School - Twitters learning resource
Scala for the Impatient - excellent "lite" version of a great book.
Scala Videos
Coursera Course - Scala course on Coursera by Martin Odersky. Only good for five weeks or so but during that time anyone can sign up and download the video lectures and materials.

Play!

Official Site
Official Docs - Excellent resource
Scala Code Samples - Some nice examples from the official Github repo.
Scala, Play! and Mongo - Blog post on writing a simple app using all those technologies.
Getting Started with Play! 2, Scala and Squeryl
Play! 2 Getting Started

Mongo

Official Website
Try Mongo - A Tiny MongoDB Browser Shell (mini tutorial included)
Little Book of Mongo (epub) - Book about Mongo
Mongly - Various tutorials and essays on MongoDB
Presentation

StackMob: Baas meets PaaS

Thu Sep 13 00:00:00 UTC 2012

StackMob Lightning Talk

Wed Sep 12 23:00:00 UTC 2012

So I finally found an opportunity to deliver a quick lightning talk on StackMob at our monthly CodeSlice sessions in work. Knocked it up today over lunch and presented it tonight. Would like to pull this one out into a more in depth and interactive look... just waiting for someone to ask :)

Securely Serving Static Files on Heroku

Mon Aug 20 23:00:00 UTC 2012

More than once I've had a need to host what is essentially static content on Heroku. For example when building a simple internal web app for my company that used an existing API or Parse. Invariably in this situation I have also needed to secure this app - nothing to drastic, no need for multiple user accounts or roles or any such nonsense (we generally run our non-essential systems on the honour system). Basic auth is more than enough.

So, static content plus basic auth. Heroku itself doesn't offer basic auth as an option (why would it?) and so you'd need to roll it into your app. Once I hosted my static stuff inside a Scala/Play! app, once a Sinatra app and the other a Rails app - variety is the spice of life. But each time I did it I felt a little dirty - it was noticeably slower serving this content from within an app and it seemed rather wasteful. So, in a bid to stamp out technical debt, I've ripped out all those web frameworks an replaced them with good old Apache.

Now anyone with any knowledge of Apache will already be able to deduce the steps required here - but I want to note it for posterity, for myself, when I forget, in about a week.

BuildPack

First things first you'll want to get your hands on a custom Heroku BuildPack - these are the things that prepare your app instance with all the software and support it needs to run.

I recommend you fork and clone the heroku-buildpack-static buildpack available from pearkes on GitHub.

.htpassword

Next you need to generate your .htpasswd file that will serve as your store for your Basic Auth credentials. The two simplest ways to generate one of these is to either use htaccesstools.com's htpasswd generator or generate on via the command line.

The simplest command to generate one is this,

htpasswd -cb <password_file> <username> <password>

For example,

htpasswd -cb .htpasswd james password

Will generate a file in the current directory called .htpasswd (the standard name for these files) with a single entry of username = james and password = password.

Updating the BuildPack

Either run the above command in the heroku-buildpack-static/conf folder or move your .htpasswd file into the that folder.

Next up you want to configure apache to authenticate against that generate password file. Adding the following lines to the httpd.conf file in the heroku-buildpack-static/conf will lock all assets under your apps site down,

    AuthType      Basic
    AuthName      "Authentication Required"
    AuthUserFile  "/app/apache/conf/.htpasswd"
    Require       valid-user

This block should be added to the configuration section starting <Directory /> i.e.

  <Directory />
      Options FollowSymLinks
      AllowOverride None

      AuthType      Basic
      AuthName      "Authentication Required"
      AuthUserFile  "/app/apache/conf/.htpasswd"
      Require       valid-user

      Order deny,allow
      Deny from all
  </Directory>

You'll notice the AuthUserFile points to the correct absolute path for our custom .htpasswd file. If you called it anything else please update this line accordingly.

Commit and push that to your forked repo.

Using the BuildPack

Finally we need to tell Heroku to use the custom buildpack. If you have yet to create an app then you can specify the buildpack during creation,

heroku create <app name> --buildpack <url to your repo>

If you have created you app already you can add the buildpack through an additional config setting

heroku config:add BUILDPACK_URL=<url to your repo>

Push your new or updated site to Heroku and watch in awe as your site is that bit more secure.

Dependencies in JavaScript: Constructor, Setter or Global

Mon Aug 20 23:00:00 UTC 2012

There has been an interesting conversation on Twitter over the last number of hours between @derickbailey and a number of other people and it all started with this tweet,

anyone have links to articles on why setter injection is evil? /cc @jbogard @jeremydmiller @jflanagan @chadmyers
— derickbailey (@derickbailey) August 20, 2012

The conversation has been around using DI and IoC in JavaScript (yes I know they are not one in the same, and I know a Container is not a necessary part of DI or IoC thank you very much). Simple answer is this - good patterns have a place in any language provided they aren't abused mercilessly. But lets get back to the discussion - why would you use constructor injection over setter injection? Is setter injection evil? Obviously no strategy is actually evil per-say - just used in the wrong place at the wrong time. So lets go back to the first question - constructor injection vs. setter injection.

Constructor Injection or Setter Injection

The simplest answer to this was one I gave @derickbailey earlier,

@derickbailey resolve mandatory dependenciesvia constructor injection, optional dependencies via setter. This is a common convention.
— James Hughes (@kouphax) August 21, 2012

This little bit of knowledge I've had since living down in the guts of Spring MVC about 5 years ago and it makes a lot of sense regardless of how those dependencies are getting in there.

Fail Early

When your code goes into an invalid state you want to fail as quickly as possible, this makes debugging much easier and prevents an avalanche of unexpected behaviour. Using constructor injection (and asserting the existence of dependencies) gives you this ability.

	var MyThingy = function(requiredA, requiredB){
		if(!requiredA) throw "Missing Dependency"	
		if(!requiredB) throw "Missing Dependency"
		
		// rest of my code
	}
	
	MyThingy.prototype.doAThing = function(){
		return this.requiredA.magic() + this.requiredB.beepboop();
	}
	
	var thing = new MyThing(); // FAIL FAIL FAIL
	thing.doAThing();

If you tried to achieve the same thing with setter injection you would be failing much later in the proceedings,

	var MyThingy = function(){}
	
	MyThingy.prototype.setRequiredA = function(requiredA){
		this.requiredA = requiredA;
	}
	
	MyThingy.prototype.setRequiredB = function(requiredB){
		this.requiredB = requiredB;
	}
	
	MyThingy.prototype.doAThing = function(){
		if(!this.requiredA) throw "Missing Dependency"	
		if(!this.requiredB) throw "Missing Dependency"
		
		return this.requiredA.magic() + this.requiredB.beepboop();
	}

	var thing = new MyThing(); 
	thing.doAThing(); // FAIL FAIL FAIL

Now imagine that the last two lines are miles apart in code-lines, you've gone and lost all context when debugging. Sure in this trivial example it would be fairly easy to resolve this but if you've gone and lost the plot and created a heavily nested object graph you'll be crying before they day is out.

Now setter injection does have it's place - for optional dependencies,

	var MyThingy = function(requiredA, requiredB){
		if(!requiredA) throw "Missing Dependency"	
		if(!requiredB) throw "Missing Dependency"
		
		// rest of my code
	}
	
	MyThingy.prototype.setOptionalC = function(optionalC){
		this.optionalC = optionalC;
	}
	
	MyThingy.prototype.doAThing = function(){
		if(this.optionalC) optionalC("doing a thing...")
		return this.requiredA.magic() + this.requiredB.beepboop();
	}
	
	var thing = new MyThing(a,b);

	thing.setOptionalC(function(s) { console.log(s) });
	thing.doAThing(); // logs "doing a thing..."

So the extension of the rule is that

Anything injected via setter injection needs to be either defaulted initially or checked for existence before use.

Readability

By following this convention, heck by following any agreed convention, you get better visibility of the expected behaviour of your code. You know what the object needs to get the job done. That feels like a poor argument but I think it still warrants a mention here.

Over Engineering and Deep Graphs

Obviously if you've over-designed your classes or it has grown out of control this approach becomes difficult to manage. I've seen something akin to this in some projects,

	var x = new CoordinatorOfAllTheThings(a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p....)

Someone has decided to make every bit of functionality it's own little class or module and so the dependency graph has become incredibly broad. In fact this might suggest they are creating a deep graph as well and simply passing the dependencies down the chain. If you have a lot of dependencies then there is something wrong with the object you are creating.

For fine grained, inter-object communication dependency injection is fine but when you are scaling up to more coarse, inter-module or component communication an evented or message based approach is almost always a better option.

	// rest of previous code
	
	MyThingy.prototype.doAThing = function(){
		EventStream.trigger("doneathing");
	}
	
	var EventStream {
		trigger: function(){ /* ... */ }
		on: function() { /* ... */ }
	}
	
	var OtherThing = function(requiredD){
		EventStream.on("doneathing", requiredD)
	}
	
	var thing = new MyThingy(a,b);
	var other = new OtherThing(function(){ console.log("stuff done"); });
	
	thing.doAThing(); // logs "stuff done"

This is a rather contrived example but it should server a point.

The Global Issue

You'll notice the EventStream object in the code above is global, you'll also notice I have skipped part of this posts title (... or Global). Lets come to that now. It is possible to achieve the same results as constructor/setter injection using global scope,

	var MyThingy = function(){
		if(!global.requiredA) throw "Missing Dependency"	
		if(!global.requiredB) throw "Missing Dependency"
		
		// rest of my code
	}
	
	MyThingy.prototype.doAThing = function(){
		return global.requiredA.magic() + global.requiredB.beepboop();
	}
	
	var thing = new MyThing(); // FAIL FAIL FAIL
	thing.doAThing();

So rather than inject the necessary dependencies in we are relying on the fact they exist in global scope. This works fine in some situations. The EventStream object above is a global hub that can be used throughout the application. I'm ok with that. The risk of global scope is that it's easy for objects to be redefined by other actors in your system so existence or behaviour is never guaranteed. So use sparingly.

Moptional

The other alternative approach would be to make everything Moptional. That is, anything mandatory should be instantiated with a default and provide setter injection for changing default behaviour.

	var MyThingy = function(){
		this.requiredA = new DefaultRequiredA();
		this.requiredB = new DefaultRequiredB();
		
		// rest of my code
	}
	
	MyThingy.prototype.setRequiredA = function(requiredA){
		this.requiredA = requiredA;
	}
	
	MyThingy.prototype.setRequiredB = function(requiredB){
		this.requiredB = requiredB;
	}
	
	MyThingy.prototype.doAThing = function(){
		return this.requiredA.magic() + this.requiredB.beepboop();
	}
	
	var thing = new MyThing();
	this.setRequiredB(new CustomRequiredB())
	thing.doAThing();

You kind of get the best of worlds here - not having to worry about default behaviour but have the ability to override this (for example in testing). But then you need to think about this. Why are you providing this sort of strategy? Are you simply hiding all implementation for an object but providing setters and getter to simply aid testing? If that's the case - STOP. Your code should be testable but you should not change your code to simply aid testing. Perhaps its time to go back and re-think your strategy? Also, for testing - it's entirely possible, given the dynamic nature of JavaScript, to redefine the internal dependencies prior to them being instantiated in your object anyway. Don't add features or behaviour that you don't actually need to deliver the solution.

Wrap Up

This was playing over and over on my ride into work this morning (I cycle I have a lot of thinking time) and I wanted to get it out there. Hopefully be useful to people asking some questions on this topic.

Oh and IoC Containers in JavaScript...... don't.

My $0.02

Backend as a Service - Part One

Tue Aug 07 23:00:00 UTC 2012

I recently wrote an article for CloudSpring (a part of the SitePoint family) around what you can expect and what you should think about when going for one of the "Backend as a Service" offerings on the market.

The article is now live.

CloudSpring: Backend as a Service Part 1

Part 2, the one where I look at whats actually out there, will be ready within the next week.

Fear of Code

Sat Aug 04 23:00:00 UTC 2012

More silly wishy washy nonsense - take what you will.

Recently Thomas Fuchs has this to say on twitter,

As a programmer, you should be afraid of code, and eliminate as much as possible.
— Thomas Fuchs (@thomasfuchs) August 2, 2012

Its great, I love this stuff and it got me thinking and I thought why not put my thoughts up here, right? After all this is what this place is all about.

Rather than describe the emotion as fear ("be afraid of code") i'd posit that, as a programmer, you should be respectful of code. Rather than eliminate as much as possible you should conserve it as much as possible - don't write it in the first place.

A while back I posted something similar on Twitter - I can't find the tweet right now but it went a little something like this,

Adding code is great, removing code is greater but not writing the code in the first place is greatest

As a developer it is very easy to fall into the trap of creating some sort of highly reusable complex masterpiece that is going to change the way we write code forever. But down that road, madness lies. Desigining too much, adding too many features, creating too many abstractions, handling every single edge case, writing too many tests - all this stuff is just extra complexity, extra code that simply muddies the solution and makes it harder for someone to understand the solution. More code doesn't solve the problem faster, more code doesn't make things easier, more code doesn't help.

In other words - writing too much code, regardless of quality is just a selfish act. That goes for 3rd party code too.

Quit it.

expect.kt - Testing Library for Kotlin

Tue Jul 24 23:00:00 UTC 2012

In a bid to delve into the language a bit further than a rake of tiny console apps I decided to do something "real" with Kotlin. With that in mind - welcome expect.kt.

It started purely as a learning venture but I think there is a certain usefulness in the library so its probably worth sharing :). Now it's worth noting that kotlin already has a number of testing statements that you can achieve a lot of this stuff with - this library gives you a few things over and above these statements

Alternative syntax. Testing is quite a personal thing and some people like to write things in different ways.
Descriptive assertions. Some would say expect(2).toBePositive() reads better than assertTrue(2 > 0) or expect(2 > 0) { true }. Again its all about taste, and variety is the spice of life.
Chained assertions. See later

Rather than some campaign to replace the inbuilt expectations and assertions this library intended to complement them.

So what is expect.kt?

Expect.kt is a set of (growing) assertions/expectations that you can use in your kotlin based tests. It gives you a number of ways to assert your expectations and gives you quite descriptive methods in which to do it.

Expect Syntax

The expect syntax is the default way to assert things,

package tests

import org.junit.Test as test
import kotlin.expectations.*

class StringExpectationTests {
    test fun testHelloWorld () {
		expect("Hello World").toStartWith("Hell")
	}
}

Most of that is boilerplate JUnit testy things but the key line is

expect("Hello World").toStartWith("Hell")

Reads quite nicely IMHO.

Should Syntax

The should syntax offers a more fluent syntax if you want to call it that, if we wanted to write the same test above using the should syntax we could do this,

"Hello World".should.startWith("Hell")

So rather than wrap your target with an expect you get and extension function to fluently assert it.

Chaining

Sometimes making a single assertion on a result isn't enough and rather than write assertTrue(...) multiple times you can use extpect.kt's chaining support.

expect("Hello World").toStartWith("Hell").and.toEndWith("rld")

Or, with the should syntax,

"Hello World".should.startWith("Hell").and.endWith("rld")

It's a bit more readable than expect after expect, right?

So thats it for now - there are a number of assertions implemented already on the useful types (Strings, Ints, Doubles, Dates, Booleans for example) and there are more planned on the way. Currently you'll be wanting to build from source (or copy/paste what you need).

Finally - A shout out to .NETs FluentAssertions where a lot of the assertions found their origin!

expect.kt.

expect.kt: Fluent assertions for Kotlin

Fri Jul 20 00:00:00 UTC 2012

Kotlin: Nullable Extensions

Tue Jul 17 23:00:00 UTC 2012

Haskell doesn't have if statements - that blew my mind the first time my old Scala mentor told me it. ifs just don't work with a strong function approach. Plus you know what - ifs are rather ugly especially when you need to wrap ALL THE THINGS in a null check. Scala has a type called Option. An Option is essentially a collection of none or one instances of a type. If everything returned an Option nothing would be null and you don't need to do any null checks (they are much more powerful than I am making out but for now this is all that is necessary).

Kotlin doesn't have the Option type but it does have two things that allow you to act like it does,

Nullable types - e.g. String vs String? , and,
Nullable Extension Functions

Remember the brief definition of an Option - it's a list of 0 or 1 instances of a particualr type and as such we can treat it a bit like a list. Now once we get functional with our badselves this makes null handling unnecessary. We can use all those nice little functional methods that you get with collections - map, forEach, filter, fold etc.

This means you're less inclined to use ifs and, more importantly, less inclined to use mutable variables which is great for building performant, scalable soltions.

Lets take a really contrived example to show you what I mean. OImagine this bit of code....

   val auth = authenticate(username, password)
   var view : View = null
   
   if (auth == null){
   		view = views.html.login("Invalid Username or Password")
   }else{
   		view = views.html.index(auth)
   }

   return view

We want to authenticate a user and return a certain view. Now yes we could refactor this down into something more bitsize but this is a very common state to find code - so bare with me for demonstration purposes.

authenticate returns a user if the username and password was correct otherwise it returns null. With our nullable extensions (by importing kotlin.nullable.*) we can reduce this into something more Scala-like,

return authenticate(username, password).map {
    views.html.index(it)
} ?: views.html.login("Invalid Username or Password")

So what we have done here is basically take an object (the user), if it exists, and transform it to another object which we return. Otherwise we return something different. This is much simpler and avoids having to create a mutable var (yes we could have avoided that in the other example but in many cases its very difficult to achieve this). In fact I am not even a fan of the elvis operator there (?:) and have a Scala-like syntax pull request open for getOrElse. A small syntactic change that I think allows you create a much neater, chainable block of code.

Thats only the tip of the iceberg on Nullable Extensions - those of you with a functional mind will be able to put these collection-like extensions to great use I reckon. Not quite Scalas Option type but good enough at providing the common features for me.

Kotlin: Extension Functions

Sun Jul 15 23:00:00 UTC 2012

In any language (that supports the concept) extension functions are nothing more than syntactic sugar. Instead of writing,

StringUtils.encrypt(mystring, "salt")

Extension functions allow you to write,

mystring.encrypt("salt")

Once you start moving into the world of chaining things get much cleaner.

encrypted = StringUtils.encrypt(mystring, "salt")
encoded = StringUtils.encode(encrypted, "UTF-8")

or worse,

StringUtils.encode(StringUtils.encrypt(mystring, "salt"), "UTF-8")

I've seen this done a few times, it happens. But with sugary sweet extension functions (Extension Methods in C#, Implicit Conversions in Scala) you could just do this,

mystring.encrypt("salt").encode("UTF-8")

Much nicer.

Kotlin, unlike crusty old Java, supports extension functions. Here is how you'd implement the methods above

fun String.encrypt(salt: String): String {
   return StringBuffer(this).append(salt)?.reverse().toString()
}

fun String.encode(encoding: String) : ByteArray {
    return this.getBytes(encoding)
}

And we can use these, as you'd expect, like so,

fun main(args: Array<String>) {
    val mystring = "james"
    val encoded = mystring.encrypt("salt").encode("UTF-8")
}

So what identified those functions as Extension Functions? Simple - the prefixing of the method name with the type String.encode. No need to create implicit wrappers (Scala), no need to create loads of static classes (C#). And yes it's possible to use generics in these methods as well,

fun <T> T?.getOrElse(t: T) : T {
    return this ?: t
}

This can be applied to any nullable type.

Literal Extension Functions

An interesting little feature that Kotlin has that makes it a bit different is the use of function literal extension functions. Basically where you can declare a function as a literal (a lambda method if you will) you can also make that literal an extension function,

So we could take our methods we wrote above and bring them right into our main

fun main(args: Array<String>) {
    val encrypt = { String.(salt: String) -> StringBuffer(this).append(salt)?.reverse().toString()}
    val encode = { String.(encoding: String) -> this.getBytes(encoding) }
    
    val mystring = "james"
    val encoded = mystring.encrypt("salt").encode("UTF-8")
}

This may seem quite unnecessary at first glance but it become quite powerful when creating DSLs - as demonstrated in the docs around the Groovy Style Builders

So Kotlin has extension functions that don't require a load of boilerplate (C#) or rather odd implicit syntax (Scala) - cool, right?

Kotlin: Here's What I Think (For Now)

Sat Jul 14 23:00:00 UTC 2012

Before I go shooting my mouth off here - a word of warning - I haven't really done anything with Kotlin. I've read the docs, played with the Kotlin 101 samples and written a few small console type apps. So take what you want from this post.

Want to hear something zany? There actually are people - real living people, with a fully working brain, many much smarter than me - that are "happy" with Java.

Seriously.

I made a move from Java to .NET about 5 years ago and have dipped back in a few times over that period. Trying new languages is a sure fire way, for some at least, to make you realise Java is not a language that makes coding fun, or productive. Java put me right of the JVM. Any language that makes woking with collections (something you'll do alot) so long winded and noisy deserves to be shunned. But this isn't a Java rant - the JVM is great but it's just marketed badly because of Java.

But all the JVM-hate that Java instilled in me 5 years ago; the last 6 months of working with Scala has replaced with love. So I've been looking around the JVM world and discovered Kotlin a new JVM based language from JetBrains. Its much closer to Java than what Scala or Clojure is (intentionally).

One main thing that Kotlin brings to the table is pragmatism. Without trying to belittle Kotlin - it's almost like CoffeeScript for Java (yep kind of like what Xtend is. Kotlin removes all those annoying necessities in Java that make your code verbose (checked exceptions, semi-colons, null reference checks etc.). It also adds things that make your life much easier (extension methods, functional collection manipulation etc.).

When you write Ruby for the first time you notice that many things just work out like you'd expect, the syntax is quite natural. The same cannot be said for Java - there is just way too much boilerplate. Kotlin helps reduce that boilerplate. So, by extension, it could be said that Kotlin makes you more productive. A person who knows Java will find Kotlin no challenge to start using - the syntax is nicely aligned, just without the noise.

Many people, when trying to get their team to adopt Scala, use the "you can use Scala like Java til you get used to it" mantra. I don't agree with that. Academically it's true but the minute you start working with other peoples Scala code - you've just gotta know Scala and when they have a slew of Implicits thrown in your Scala as Java stuff goes out the window. Kotlin, on the other hand, has some of the nice functional patterns from Scala but none of the "magic-to-a-newcomer" features that can make grown men weep. Now the Kotlin docs themselves even go as far as stating

If you are happy with Scala, you probably don't need Kotlin.

But I think they are selling themselves short - I think plenty of people use Scala but could easily use Kotlin in its place and find themselves in a better place (easier upskilling, less cryptic codebase).

Anyway this is a bit of a ramble, what I'm saying is - Kotlin has some really rather nice features (Groovy Like DSL Builders, Pattern Matching, the when expression, lambdas, Kotlin to JavaScript compilation(!) etc.) and people happy with either Java OR Scala should give it a go.

Azure Website Workflow for Bosses

Wed Jul 11 23:00:00 UTC 2012

So consider my indefinite hiatus suspended definitely. I suck at quitting! But we wont dive into the whys and wherefores around that now.

I've a confession to make, the "new" Azure, I like it. Yeah I do, s'nice. I like the way it blends PaaS with IaaS (though I do feel that adding 1st class node.js and PHP support is a bit of a tacked on marketing ploy). So you dont have the "Add-On" model you get from Appharbor and Heroku but thats fine because this blending of "as a Services" gives you an alternate approach. For a start you can just use the external services without them having to be "Add-Ons". Sure you lose out on the nice automatic integration points for managing your services but it's not the end of the world and what happens when you want something that isn't available as an Add-On or worse not available as a cloudy SaaS thing? Well I guess you need to go off to EC2 or similar and host your own (assuming you want to stick with the cloudy infrastructure). But with Azure you also get another option - you want to use Mongo or some huge enterprisey platform that isn't SaaSified? Stick it on an Azure VM. Everything managed under one cloud provider. Thats such a nice thing to have at your disposal.

But thats not the point of this post. Most posts out there cover a workflow for Azure WebSites that make use of the portal to provision new sites (and then usually git to deploy) but that's weak sauce (the portal part at least - gits still aces). No, no no - You want hyper-productivity, you want automated provisioning, you want to be hip; node.js hip; and you want to feel like a boss hanging out on your command line while other "lesser people" click stuff with a silly little rodent. I just read your mind, didn't I? Well then - you got that with Azure.

Setting Up

First things first you need to install a few things.

Installing node.js + NPM

Download

The tools are built and deployed in node via NPM, the node package manager so just run the one click installer and you'll be all setup.

Installing Azure CLI Tools

Install these via NPM so fire up your terminal/command prompt/whatever and type

npm install azure -g

Then let NPM install all the dependencies and what not (the -g option just tells NPM to install the tools globally so they are accessible from the command line)

Adding your account

Next up you need to tell azure what account to use so fire this command into you command line

azure account download

You'll then be taken to the Azure website and it should (assuming you're logged in already) download a publishsettings file. Next up you need to import this file into your azure toolchain.

azure account import YOUR_DOWNLOADED_FILE.publishsettings

After some scrolling text you'll be all set up and ready to deploy. You'll be asked to delete the file you downloaded - probably best to do this :)

Creating the app

OK then lets create an app - what we create is irrelevant here so lets just create the default ASP.NET MVC website and use that as our sample app. Remeber though, if you're a cutting edge awesome-addict to switch from .NET 4.5 to 4 or else you'll get some issues when deploying your site to Azure.

UPDATE : The next few steps can be condensed, thanks to the feedback from Glenn Block (@gblock),
@kouphax oh dude, you don't need all those steps! Do "azure site create my site --git" It will create the repo for you and add the remote!
— Glenn Block (@gblock) July 16, 2012
This sets up your git repo during creation of the site, adds the remote endpoint and even generates a .gitignore file (which you need to edit yourelf for .NET projects. WIN!!!!
Anyway.... on with the old post.

Once you've created you app and saved it then you'll want to intialise an empty repo at the root of the solution,

git init

Then we need to create a site on Azure we can push to.

Creating a new site

Back to the command line now,

azure site create SITE_NAME

If we execute this, again at the root of our solution, it will do a number of things,

Validate the request (site name availability and what not)
Create a new site on Azure
Add a new git remote to your .git\config file

Pushing site

As I've said the last command actually added a remote repo to our git configuration so if we do the usual gitty stuff (you'll want to, for neatness sake, set up your .gitignore but we can skip this for brevity),

git add -A
git commit -m "Boring commit message"

Now we push,

git push azure master

Done!

Viewing your site

You could just browse to your new site but thats just not hip enough, hell no. Instead lets launch it from the command line...

azure site browse

That will launch our new site without having to touch a silly old, out of date rodent.

Wrapping up

Ok so thats the basic workflow for using CLI tools to manage Azure sites. You can do this with most of the Azure services (VMs, Certs, Cloud Services etc.) aswell.

So apart from making you look all hackery working in the terminal it does have other uses, specifically around the fact terminal command can be easily executed in a headless automated environment. Think of creating new sites as part of your CI's deployment pipeline, or as part of some disaster recovery process. The possibilities are endless... heck even write your own UI for managing sites or deploying new sites/VM/whatever!

So lets see -

CLI hackery coolness... ✓
Node.js hipster bragging rights... ✓
Azure website created and deployed with about 6 commands... ✓

Not bad for a few minutes work!

Technology in the Cloud

Wed Jul 11 23:00:00 UTC 2012

The Cloud. As much as the term has been pummelled into meaningless marketing sputum by the guys in expensive suits and fake smiles it does exist and is distinct from a typical "throw all your stuff in a data center" setup.

The Cloud is distinct, as well, in terms of how you think about your solution architecture. The typical old school approach would be to just create big monoliths to process your data but that doesn't work out so well in the cloud. Of course the considerations for the cloud aren't new, hell no - things like SOA have been around for an age, but if you want to harness the benefits of the cloud as the internet intended you will be narrowing your options. No more just cobbling common bits together and crossing your fingers.

So fine - architecting for the cloud has some special considerations but thats not what I want to talk about - there are better people than me to cover that.

What I want to talk about is use of technology within that architecture. I've never been a fan of the one-size-fits-all-painful-or-not technology stack driven almost solely by a need to avoid unnecessary education of developers. And, now I feel the cloud is strengthening my view that a default stack is more trouble than it's worth. More importantly it's really adding momentum to the whole polyglot architecture, or simply using the right tools for the job.

Let me explain. Any default stack, by definition, needs to cater to many user cases and as such brings a certain amount of sacrifice with it. Sacrifices come in many forms but usually in performance, memory footprint and/or verbose configuration. In the cloud you are, in essence, in a Pay-As-You-Go model, every clock cycle and every bit of storage costs. Now that heavy ORM with its less than ideal queries and slow relationship mapping is starting to cost you money EVERY request. In fact, it's probably backed by that storage heavy relational database for all your non-relational data. Same for that rather large web framework you've got there.... every time to deal with a spike and scale that badboy out and you're having to use medium instances rather than small. That SOAP based webservice that requires triple the bandwidth to service a request... ugh you get the point.

In the cloud the default stack can easily bleed money.

I guess this is less about the default stack and more about just using the right tools to get the job done in the most effective manner - afterall you're paying for this stuff now. You kind of were before but in such a coarse grained manner it didn't matter - it does now.

In summary - the cloud has started to make technology decisions more accountable and we need to ensure that those decisions aren't costing our stakeholders more money than necessary.

PS.

I'm not going to sit here and preach about the which specific technologies to use but come on - if your storing Session State in a Oracle database, switch to Riak, or Redis. If you're building a nice scalable API then use something like Scala which promotes immutable state and concurrency out of the box over something like vanilla Java (or perhaps just a better framework Spring-WS vs Dropwizard for example). JSON over XML or a binary format (Protobuf for example) for machine to machine communication. A small node.js web site over ASP.NET Webforms for simple sites. The list goes on. That is to say consider this advice against the other factors driving technical decisions - dont JUST go for the smallest, lightest tools because it MIGHT be cheaper. Do the research.

Finally, perhaps you've got another IT team needing to support this new solution? Bring them along this journey too, educate them as you educate yourself, dont just throw the finished thing over the wall with a 1 week handover and a pat on the back.

Don't be afraid - it's a super fun world we live in!

Agile Practices

Sun Jul 08 00:00:00 UTC 2012

Ruby by Example

Sun Jul 08 00:00:00 UTC 2012

Git Basics

Sun Jul 08 00:00:00 UTC 2012

Ruby on Rails

Sun Jul 08 00:00:00 UTC 2012

Quitsies, Well Sort Of

Mon Jul 02 23:00:00 UTC 2012

You know when you start to question why you write these things it's probably time to quit, or at least take a break.
~ me, just there now

This more of an FYI to anyone who may read this blog than a horrible diatribe about how I am wasting my life with meaningless exercises or technologies I'll never get to use to a bunch of people that don't really care, what would be the point? Plus - it's probably not even true.

So rather than pottering on about cool and interesting technologies that will make everyones (read, developers) life infinitely better 100% guaranteed I've decided to devote my time to,

Working on some startup-py type things that have been milling around the back of my head for some time, and,
Generally getting stuff done

Yep, like a hipster band, this little testament to bike-shedding is going on indefinite hiatus.

[Quick and Dirty, Super Ugly,] Starter for 10 - Dropwizard

Mon Jul 02 23:00:00 UTC 2012

So once again I'm writing Java without a gun to my head, and once again I'm finding it enjoyable. Don't get me wrong Java as a language feels very long winded, old and cumbersome but the frameworks and stuff that exist around it actually make those issues, less prominent, so much so I think I could easily live with the noise that Java brings.

Anywho.

Dropwizard is a framework for creating very simple ReSTful Web Services in Java. Not just that but it also bundles some nice best practise in with it as well. Things like pluggable health checks, logging, configuration, content negotiation etc. And, because it's fairly lightweight its fast. Very fast.

In fact you can think of Dropwizard as a collection of pre-existing tools glued together with some simple features to create a cohesive stack for HTTP+JSON web services. You've got Jetty, Jersey, Jackson, Metrics, Logback, JDBI and more. Dropwizard give you the glue that binds these along with a simple pipeline that makes use of these technologies.

Simple Todo

I had intended to do something a little more substantial but I am about to make a small annoucemnet that has kind of effectivley stopped me expanding this app much further. So lets create a simple Todo App.

pom.xml

As with most Java projects lets start with the pom.xml which is effectively our project definition. I'll not show you my entire POM, at least not on the first date, but let me reveal the important parts.

Beause it's awesome I want to include WebJars so I add my repository,

    <repositories>
        <repository>
            <id>webjars</id>
            <url>http://webjars.github.com/m2</url>
        </repository>
    </repositories>

Next I add my dependencies,

Dropwizard Core which has everything you'll need for this
jQuery from the WebJar repository

    <dependencies>
        <dependency>
          <groupId>com.yammer.dropwizard</groupId>
          <artifactId>dropwizard-core</artifactId>
          <version>0.4.2</version>
        </dependency>
        <dependency>
          <groupId>com.jquery</groupId>
          <artifactId>jquery</artifactId>
          <version>1.7.2-1</version>
        </dependency>
    </dependencies>

I also add the shade plugin which will let me build a Fat Jar (a jar with all my dependencies and what not in it).

    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>1.6</version>
        <configuration>
            <createDependencyReducedPom>true</createDependencyReducedPom>
            <filters>
                <filter>
                    <artifact>*:*</artifact>
                    <excludes>
                        <exclude>META-INF/*.SF</exclude>
                        <exclude>META-INF/*.DSA</exclude>
                        <exclude>META-INF/*.RSA</exclude>
                    </excludes>
                </filter>
            </filters>
        </configuration>
        <executions>
            <execution>
                <phase>package</phase>
                <goals>
                    <goal>shade</goal>
                </goals>
                <configuration>
                    <transformers>
                        <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
                        <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                            <mainClass>se.yobriefca.TodoService</mainClass>
                        </transformer>
                    </transformers>
                </configuration>
            </execution>
        </executions>
    </plugin>

Configuration

Dropwizard has the ability to map configuration options (from a YML or JSON file) into concrete classes that can be accessed by your services. So first I created a todo.yml at the root of the project (beside the pom.xml) with some basic configuration

http:
    rootPath: "/app/*"

This configuration tells Dropwizard that my services are located at a url starting with /app - eg /app/todo. This is necessary as I want to create an asset later that maps to the root url / that is the default of Dropwizard. At this point I haven't needed any of my own configuration but if I did I could create a POJO that maps to proeprties in this file. The official guide has more on this. For now I will create an empty Configuration class in case I want to extend it later (yeah I know YAGNI YAGNI KISS YAGNI - shutuppayourface!)

package se.yobriefca;

import com.yammer.dropwizard.config.Configuration;

public class TodoConfiguration extends Configuration {
}

Service

The service is the core class in our solution. It's the thing that has the main as an entry point. The thing that "set us up the resouces" and registers healthchecks and controls our service lifecycle. You'd think there would be loads of code here? Right? Wrong!

package se.yobriefca;

import com.yammer.dropwizard.Service;
import com.yammer.dropwizard.bundles.AssetsBundle;
import com.yammer.dropwizard.config.Environment;
import se.yobriefca.resources.TodoResource;

public class TodoService extends Service<TodoConfiguration> {
    public static void main(String[] args) throws Exception {
        new TodoService().run(args);
    }

    private TodoService() {
        super("todo");

        addBundle(new AssetsBundle("/assets", "/"));
        addBundle(new AssetsBundle("/public", "/public/"));
    }

    @Override
    protected void initialize(TodoConfiguration configuration, Environment environment) {
        environment.addResource(new TodoResource());
    }
}

So all services extend Service and have a Configuration type (FYI these config types can be nested so its not as restrictive as I first thought). Basically what we have is a main method that create an instance of our service and runs it. Upon creation we add two bundles to the service. A bundle is bascially a bit of functionality and in this case we create two AssetBundles which can be used to serve static content (embedded resources) upon request to a specific URL (think things like CSS, JavaScript, HTML etc.) You'll notice I am mapping 2 things bundles here and this is because I need to map to the WebJars directroy as well. So when I got to localhost/index.html it will match the url pattern in the first AssetsBundle and look for a file src/main/resources/assets/index.html.

Finally we have the initialize method that allows us to decare mutliple resources and healthchecks against our environment. We also get handed our configuration class we decalred in advance.

A bit about healthchecks

Healthchecks are little classes that perform some sort of test on a part of your solution. For example we could have a database connectivity check. These healthchecks are made available on the admin section that is bound to another port when your app starts.

Now I didn't define any for this example and for that I feel absolutely terrible. Seriously I kind of feel sick thinkging about it. And whats worse Dropwizard kindly reminds me everytime I start the app - using LOTS of exclamantion marks and CAPS!!! IRL you should really not follow my example here.

Resources

Resources are resources as they are identified in ReSTy things. For example a User is a resource or an Order etc. These things will respond to HTTP requests matched against the URL and the HTTP Method used. In my example I have defined a resource Todo that can be used to list, create and delete (I didn't need anything else for this).

@Path("/todos")
@Produces(MediaType.APPLICATION_JSON)
public class TodoResource {
    private static final ArrayList<Todo> todos = Lists.newArrayList();

    @GET
    public Collection<Todo> list() {
        return todos;
    }

    @POST
    public Todo add(@Valid Todo todo){
        todos.add(todo);
        return todo;
    }

    @DELETE
    @Path("/{id}")
    public Todo delete(@PathParam("id") final LongParam id){
        Todo todo = null;
        List<Todo> todoCopy = Lists.newArrayList(todos);

        todos.clear();

        // loop past all items in the list and replace the changed one
        for (Todo element: todoCopy) {
            if(element.getId() != id.get()){
                todos.add(element);
            }else{
                todo = element;
            }
        }

        return todo;
    }
}

Again not a great deal of code and what code that is there is fairly straightforward - some would say boilerplate. We use annotations to map our HTTP Methods @GET, @POST and @DELETE as well as a @Path(...) annotation to specify what url this resource resonds to. Another nice feature we can see annotated here is we get both validation (via the @Valid annotation which is based on Hibernates Validation framework) and JSON model binding via Jackson and model annotations (get to that in a minute).

For this simple demo I am storing a list of Todos in an ArrayList in memory (top of the class).

Model

The model is, bar all the annotations to enable serialisation, fairly POJO-like,

public class Todo {

    // Not So Universally Unique!
    private static final AtomicLong NSUUID = new AtomicLong();

    @JsonProperty
    private final Long id;

    @NotEmpty
    @JsonProperty
    private final String content;

    @JsonCreator
    public Todo(@JsonProperty("content") String content) {
        this.id = NSUUID.incrementAndGet();
        this.content = content;
    }

    public String getContent() {
        return content;
    }

    public Long getId(){
        return id;
    }

}

I've put this in more for completeness as there is little involved in this.

index.html

I am not going to create some elaborate over architected front end here - just a few lines of JavaScript (ignoring the rather large jQuery library included of course) and not styling.

<!DOCTYPE html>
<html>
  <head>
    <title>Demo</title>
  </head>
  <body>
    <div>
      <input type="text" id="task" /> <button id="addTask">Add Task</button>
    </div>
    <ul id="todos"></ul>
    <script type='text/javascript' src='/public/jquery.min.js'></script>
    <script>
      $(function(){
        // load and replace the list
        $.getJSON("/app/todos", function(todos){
          for(var i = 0; i < todos.length; i++){
            $("#todos").append(
              "<li><input type='checkbox' data-id='" + todos[i].id +"'/>" +
                todos[i].content +
              "</li>"
            )
          }
        });

        // manage adding a new task
        $("#addTask").click(function(){
          $.ajax({
            url:         "/app/todos",
            type:        "POST",
            data:        JSON.stringify({ content: $("#task").val() }),
            contentType: "application/json; charset=utf-8",
            dataType:    "json",
            success:     function(data, textStatus, jqXHR){
              $("#todos").append(
                "<li><input type='checkbox' data-id='" + data.id +"'/>" +
                  data.content +
                "</li>"
              )
            }
          })
        })

        $("#todos").delegate("input[type=checkbox]", "click", function(){
          var chk  = $(this),
          item = chk.parent("li"),
          id   = chk.data("id");

          $.ajax({
            url:         "/app/todos/" + id,
            type:        "DELETE",
            contentType: "application/json; charset=utf-8",
            dataType:    "json",
            success:     function(data, textStatus, jqXHR){
              if(data.id === id){
                item.css({"text-decoration": "line-through"});
                chk.remove();
              }
            }
          })
        })
      })
    </script>
  </body>
</html>

And thats it. Oh wait I also added a boot command that peforms the maven packing and starts the jar.

mvn package;java -jar target/todo-0.1.jar server todo.yml

Just run ./boot (assuming you've chmod'd the file to be executable) and you'll have, assuming I (or you) haven't made any mistakes, a Todo service running on port 8080 (with admin/diagnostics stuff on port 8081).

As always the example code is available on Github. Sorry for the rushed nature of this post, I'll hopefully keep tweaking the example on Github till it's a bit nicer - perhaps even with tests and comments (YEAH RIGHT!!!!)

In short - Dropwizard - simple ReST services, very little config, very nice.

Super Basic Trello API Wrapper for Scala and Play

Sun Jun 24 23:00:00 UTC 2012

So I am currently doing a bit of work with using Trello as a kind of data store and decided to write a litle app in Play! (the Scala variety of course). There didn't appear to be a library out there to handle some of the basic Trello stuff so over the past 20 minutes I put one together.

import play.api.libs.ws.WS

case class Trello(key: String, token: String) {

  val host = "https://api.trello.com"

  def get(uri: String, query: (String,String)*) = request(uri, query:_*).get()
  def post(uri: String, query: (String,String)*) = request(uri, query:_*).post("")
  def put(uri: String, query: (String,String)*) = request(uri, query:_*).put("")
  def delete(uri: String, query: (String,String)*) = request(uri, query:_*).delete()

  private def request(uri: String, query: (String,String)*) = {
    WS.url(host + (uri.startsWith("/") match {
      case true => uri
      case false => "/" + uri
    })).withQueryString(query ++ Seq("key" -> key, "token" -> token): _*)
  }
}

Yep it's very basic and all but the get is untested (it's all I've needed so far). But in the spirit of shipping early this is a good start. The class makes use of Play!s uberflexible WS class and returns Promise[Request] so you can use it like so (just an example)

A simple API object to wrap the Trello class instance to load cards and map them into my expected objects

object Api {

  val api = Trello(KEY, TOKEN)

  implicit object CardFormat extends Format[Card] {
    def reads(json: JsValue): Card = Card(
      (json \ "name").as[String]
      (json \ "due").as[String]
    )
    def writes(card: Card): JsValue = throw new NotImplementedException()
  }

  def getPastTalks = {
    api.get("/1/lists/_____", "cards" -> "open", "card_fields" -> "desc,name,due").map { res =>
      (res.json \ "cards") match {
        case JsArray(cards) => cards.map(_.as[Card])
      }
    }
  }
}

And calling this bad boy from a controller.

object Application extends Controller {
  def index = Action {
    Async {
      Api.getPastTalks.map { talks =>
        Ok(views.html.index(talks))
      }
    }
  }
}

So - lots of work to do around this - I want to move away from a thin wrapper to something a bit more robust with strong classes for the different entites.

Also if you want to find out where to get the API keys etc. go read the decent API docs from Trello.

Two Hidden Gems in Play 2's Template Engine

Sat Jun 16 23:00:00 UTC 2012

Poking around some Play 2 samples today I came across some little bits of sugar. I haven't seen mention of them in other places (they must be well hidden) but they are two very common things I've implemented in less elegant ways in the past. That was reason enough for me to throw a quick post up about them.

Prepare to have your minds blown. Well not blown - gently nudged probably.

String.when

"somestring".when(predicate: => Boolean) allows you conditionally spit out the value of the string it is called on (otherwise returning and empty string).

"returned_string".when(true)      // returns "returned_string"
"nonreturned_string".when(false)  // returns "" (empty string)

So in the context of views what is this good for? Lots of things but the most clever use for this little function is conditional css. Imagine you've got a block of HTML that represents a message area. These messages can be either good or bad and so we need to add the css class of message-bad depending on an argument passed in to the view. A very quick and dirty way of handling this would be as such

@(message: String, successful: Boolean)

<div class='message @if(!successful){ "message-bad" }'>
  @message
</div>

Grand, fine. It's a bit noisy but it works. when() makes this a bit neater IMHO.

@(message: String, successful: Boolean)

<div class='message @("message-bad".when(!successful))'>
  @message
</div>

Yep the line is the same length but it looks more like a single block of conditional code, it flows much better and, I think at least, is much more readable.

Date.format("<date-format>")

new java.util.Date().format(pattern: String) allows you to specify how a date should be formatted in the rendered HTML. It's essentially shorthand for the noise-fest that is SimpleDateFormat

new java.text.SimpleDateFormat(pattern).format(date)

In the past I've added transient properties to models for displaying the date, or written my own methods on those models. Both of those approaches should have had me taken out into a field and shot. format makes things much more straightforward,

@(importantDate: java.util.Date)

<p>
  @importantDate.format("dd-MM-yyyy")
</p>

And the output will be along the lines of "29-01-2013" or whatever the date was.

OK so neither of these are ground breaking or earth shattering (wait they mean the same thing really dont they???) but they are neat to enough to warrant a shout-out here. Anyone else have any little nuggets in Play 2 they want to share?

MonoTouch - C# + iOS = Good Times

Fri Jun 08 00:00:00 UTC 2012

.NET On a Diet

Fri Jun 08 00:00:00 UTC 2012

Monotouch: C# + iOS = Good Times

Wed Jun 06 23:00:00 UTC 2012

Slides

Code

https://github.com/kouphax/monotouch-samples-ndc2012

Description

MonoTouch allows you to harness all those C# skills and most of those handy .NET/Mono libraries to create truly native apps for iOS with a sprinkle of cross-platform support. But why not just use objective-c you cry? Choice is nice, so is C# and so is the range of libraries available for mono!

In this talk I will dive in to what makes MonoTouch so compelling for .NET developers wanting to break into iOS development and even how these skills can be harnessed to create cross platform native applications for iOS, Android and WP7 through a shared codebase.

.NET on a Diet

Wed Jun 06 23:00:00 UTC 2012

Slides

Description

There are plenty of framework options in .NET. We have the big mainstream web frameworks like ASP.NET WebForms and MVC, the ORM abstractions and DataMappers in the form of Entity Framework, NHibernate and iBatis and services such as WCF requiring large amounts of careful configuration to get right. Unfortunately these frameworks can get in the way, abstractions breakdown and leak and eventually we end up fighting against the frameworks limitations.

This session will take a look at the feature-fit lightweight alternatives and how they can be used to compose a technology stack that is cruft-free and fit for purpose.

Mobile, How Do I Develop Thee? Let Me Count the Ways (Long Version)

Wed Jun 06 23:00:00 UTC 2012

Slides

Description

Mobile development is a bit like writing web apps back in the browser war days - full of decision anxiety, volatility and platform specific issues. There is no one true answer yet most people think there should be. This talk will cover the popular(ish) options from native all the way to web solutions (and all the mashup magic in-between). Oh yes and some opinion as well.

This is the longer more up to date version of the previous talk.

Mobile, How Do I Develop Thee? Let Me Count the Ways

Wed Jun 06 00:00:00 UTC 2012

5 Years

Fri May 25 23:00:00 UTC 2012

So I am sitting here, here being London - 315(ish) miles away from my family, halfway through the NHS Hack Day and having a bit of a personal crisis. I often have these little episodes but this one has been languishing in my dusty old head for a good while - refusing to go away, nagging me every time I have a chance to ponder things. It's the same old thing - why am I doing any of this stuff I do? I don't generally get a great amount of opportunity in work to really stretch my creative urges. That sort of thing happens on my own time and therefore given my other responsibilities (my wife, family and lifes associated activities) is done under a fairly tight deadline. Add this to the fact I bore easily of things that aren't really benefiting people immediately - I tend to flit, easily, between little unfinished projects. But wait.... I did it again. You see back there, about 2 sentences ago I just sidelined my entire life. Thrown between two looming brackets I relegated my wife and family to a byline yet again. Oy vey.

And this brings me, neatly enough, to the point of this entire ramble.

Five years ago I married my awesome wife and in that time I've done a heap of things - relocated twice, had two kids bought a house, got promoted, pissed everyone off, seen a few countries and bought/traded three cars (which get smaller every time oddly enough). And through all this hopping around my wife has been the one true constant. She thinks, most of the time, I don't care about her - after all I'm in London participating in a hackathon on our 5th wedding anniversary because I stupidly didn't put two and two together and notice the clash. So yeah it may damn well seem like she comes in a close second to my hobby/career stuff (you know the stuff I already mentioned was pulling into crisis mode). But it's not true. I can't help the fact I am "emotionally retarded", as my sister once called it, but perhaps once in a while I really should let her know that she means more than she thinks she does. And here it is. A little bit of emotional letting if you will.

Dear Emma - happy anniversary, I hope we have plenty more, thank you for putting up with me, you're aces.

Starter for 10 - Scala, Play 2 and Mongo

Mon May 07 23:00:00 UTC 2012

This post is going to cover the whole "getting up an running" for a simple Scala stack - namely Scala, Play 2 and Mongo. 2012 has been the year of the polyglot for me. I've been working on projects with .NET, Node.js, Ruby and Scala in the mix. I've seen how Rails can really boost developer productivity (compared to the likes of ASP.NET MVC or other, less opinionated, frameworks) but I must say - In terms of developer productivity, at least for me, nothing has come close to the Scala/Play/Mongo combo. I hadn't touched functional programming since SML in university but I found Scala allowed me to express myself in a very terse manner. Combine that with Play and its straightforward, uncomplicated approach to building web apps and Mongo with its schemaless nature then its productivity++ all the way.

It's worth pointing out right away that James Ward already has a similar post and a (currently Java only) tutorial. These are much better but they dont cover the Mongo/Caabah/Salat side of things (not that it is particularly difficult) so at least I've got a leg to stand on.

Installing

Versions used in this post

Scala (2.9.1)

Play (2.0.1)

Mongo (2.0.4)

I'm going to assume that you are capable of installing software (by following basic instructions) so I won't waste your time covering this. Here are some links to kick you off,

Scala
Play 2 (it just goes on your PATH)
Mongo

Alternatively if you are on OSX - Homebrew has got everything you need at the time of putting this together.

Generating your Application

Play is a command line tool and can be used to generate your basic app skeleton,

play new <project name>

Just fill in the project name and you will be guided through the basic generation of the project, e.g.

--> play new sampleapp
        _            _
  _ __ | | __ _ _  _| |
| '_ \| |/ _' | || |_|
|  __/|_|\____|\__ (_)
|_|            |__/

play! 2.0.1, http://www.playframework.org

The new application will be created in /Users/kouphax/Projects/temp/sampleapp

What is the application name?
> sampleapp

Which template do you want to use for this new application?

  1 - Create a simple Scala application
  2 - Create a simple Java application
  3 - Create an empty project

> 1

OK, application sampleapp is created.

Have fun!

Indeed - fun you shall have.

Dependencies

Working with Mongo directly is all well and good but it can be made a bit simpler with a few dependencies. Namely Casbah and Salat. Casbah is the official Scala Toolkit for MongoDB and Salat allows you to map to and from the generic Mongo object classes into more concrete case classes.

To add the dependencies you need to update the Build.scala file in the generated project directory. Firstly you need to update the appDependencies to create the dependency on the new two libraries,

val appDependencies = Seq(
  "com.mongodb.casbah" %% "casbah" % "2.1.5-1",
  "com.novus" %% "salat-core" % "0.0.8-SNAPSHOT"
)

Then you'll need to add the resolvers to the main project declaration,

val main = PlayProject(appName, appVersion, appDependencies, mainLang = SCALA).settings
  resolvers ++= Seq(
    "repo.novus snaps" at "http://repo.novus.com/snapshots/")
)

UPDATE: Since this article is a bit old but still getting plenty of hits I have noticed that people are having issues resolving dependencies on salat and casbah - this is because of a reporsitory change. Simple fix just add the following line to the resolvers Seq above,

"casbah" at "https://oss.sonatype.org/content/groups/scala-tools/"

This should be enough to get you going with the rest of this article however there will be some major changes expected in Scala 2.10 that may well fundamentally alter the need to use Salat etc. Be warned.

And thats the project set up with all its dependencies ready to hack on. So lets drive a spike through this stack and see what it looks like shall we? Yes why not, tallyho old bean!

The Model & Data Access

First things first lets create our model. So we are going to write a very basic registration app. People "register" on the site, for fun or something, I dunno as our stakeholders - I just write the code.

Our model is super simple,

case class Registration(
  username: String,
  password: String,
  confirm:  String,
  realName: String
)

Now to the interesting part. To persist this in a Mongo data store we need to make use of the two libraries we imported previously,

import com.mongodb.casbah.Imports._
import com.novus.salat._
import com.novus.salat.global._

object Registrations {
  val registrations = MongoConnection()("sampleapp")("registrations")

  def all = registrations.map(grater[Registration].asObject(_)).toList
  def create(registration: Registration) {
    registrations += grater[Registration].asDBObject(registration)
  }
}

First things first we create a connection using the MongoConnection class and grabs the registrations collection from the sampleapp store. Now it's worth noting that provided Mongo is up and running we don't actually need to create either the store or the collection - the act of acting on a collection is enough to create both.

Next up I defined two methods all and create.

all simply pulls out all the collection data using Casbah, performs a map over the collection (which is a collection of MongoDBObject/DBObjects) and uses the grater object provided by Salat to map from these generic objects to our Registration case class.
create simply adds an element (cast from a Registration class to an acceptable DBObject) to the collection.

Now isn't that a nice terse data access layer? Yes it is. You want to make changes? Go ahead - the fear of change is gone or much reduced compared to the typical approach.

Controllers

Next up we want to add a register controller and update the existing Application controller.

object Application extends Controller {
  def index = Action { implicit request =>
    Ok(views.html.index(Registrations.all))
  }
}

The only difference here is the Registrations.all so we can list the registrations in the UI. Lets jump over the the register controller

object Register extends Controller {
  def registrationForm = Form(
    mapping(
      "username" -> nonEmptyText,
      "password" -> nonEmptyText,
      "confirm" -> nonEmptyText,
      "realName" -> text
    )(Registration.apply)(Registration.unapply)
      verifying("Passwords must match", fields => fields match {
        case Registration(_, password, confirmation, _) => password.equals(confirmation)
      })
  )

  def index = Action { implicit request =>
    Ok(views.html.register(registrationForm))
  }

  def register = Action { implicit request =>
    registrationForm.bindFromRequest.fold(
      form => BadRequest(views.html.register(form)),
      registration => {
        Registrations.create(registration)
        Redirect(routes.Application.index()).flashing("message" -> "User Registered!")
      }
    )
  }
}

OK thats a lot to take in so lets break it down a bit.

The Form

def registrationForm = Form(
  mapping(
    "username" -> nonEmptyText,
    "password" -> nonEmptyText,
    "confirm" -> nonEmptyText,
    "realName" -> text
  )(Registration.apply)(Registration.unapply)
    verifying("Passwords must match", fields => fields match {
      case Registration(_, password, confirmation, _) => password.equals(confirmation)
    })
)

The Form concept in Play is a bit like a ViewModel of sorts. It defines validation and mapping rules and allows you to pass a structured object between the controller and the view (inluding error information etc.). In fact you aren't even obliged to bind this to a class, they can exists as themselves. My example above shows you how to bind to and from our Registration class as well as some field and crossfield/global validation.

Model Binding

registrationForm.bindFromRequest.fold(
  form => BadRequest(views.html.register(form)),
  registration => {
    Registrations.create(registration)
    Redirect(routes.Application.index()).flashing(
      "message" -> "User Registered!"
    )
  }
)

This little functional approach allows us to bind a request to our registrationForm and react to invalid input. In the case of an invalid bind the first argument into fold is executed which returns a bad request with an instance of the form. In the case of the happy path we simply push our registration instance into the data store and redirect to the index action along with a flash message.

Flash

Flash is like a carefully curated session variable that exists only for the next request. This allows you to pass messages or some very transient data between requests. This is also accessible in the view which is handy.

Routes

Routes in Play are written into the config/routes file and this is used as a basis to generate reverse routes (this gives is us statically typed routes via the routes object).

# Routes
# This file defines all application routes (Higher priority routes first)
# ~~~~

# Home page
GET     /                           controllers.Application.index
GET     /register                   controllers.Register.index
POST    /register                   controllers.Register.register

# Map static resources from the /public folder to the /assets URL path
GET     /assets/*file               controllers.Assets.at(path="/public", file)

So I added two new routes for register that allows us to view and submit our registration form.

The Views

Right lets do something a little neat - remember I mentioned James Ward earlier? Well he curates a little project called webjars which allows us to add client side resources as managed dependencies to our project. So lets prettify our UI with a bit of Twitter Bootstrap love.

First thing we want to do is update our Build.scala file with webjars resolver and our bootstrap dependency

val appDependencies = Seq(
  "com.github.twitter" %  "bootstrap"  % "2.0.2",
  // other deps...
)

val main = PlayProject(appName, appVersion, appDependencies, mainLang = SCALA).settings(
  resolvers ++= Seq(
    "webjars"          at "http://webjars.github.com/m2",
    // other resolvers...
  )
)

Then in our main.scala.html we can reference the bootstrap assets in the normal way,

<link rel="stylesheet" media="screen" href="@routes.Assets.at("stylesheets/bootstrap.min.css")">

Index (index.scala.html)

Here we simply list the current registered people,

@(registrations: List[Registration])(implicit flash: Flash)
@main("Super Simple Sample") {
  @if(flash.data.contains("message")){
    <div class="alert alert-success">@flash.get("message")</div>
  }
  <a href="@routes.Register.index" class="btn btn-primary">Register</a>
  <table class="table table-bordered">
    <thead>
      <tr>
        <th>Username</th>
        <th>Real Name</th>
      </tr>
    </thead>
    <tbody>
      @registrations.map { registration =>
        <tr>
          <td>@registration.username</td>
          <td>@registration.realName</td>
        </tr>
      }
    </tbody>
  </table>
}

The interesting thing about Play 2 views is that they are just Scala. Yep neat. In fact any .NETters will feel quite at home as they are reminiscent of Razor views too.

Register (register.scala.html)

@(registrationForm: Form[models.Registration])(implicit flash: Flash)

@import play.api.i18n._
@import views.html.helper._

@input(field: Field, label: String, fieldType: Symbol = 'text) = {
  <div class="control-group @if(field.hasErrors) {error}">
    <label class="control-label" for="@field.id">@label</label>
    <div class="controls">
      <input type="@fieldType.name" value="@field.value" name="@field.id" />
      @if(field.hasErrors){
        <span class="help-inline">
          @Messages(field.error.head.message)
        </span>
      }
    </div>
  </div>
}

@main("Super Simple Sample") {
  @form(action = routes.Register.register, 'class -> "form-horizontal") {
    <fieldset>
      <legend>Registration</legend>
      @registrationForm.globalError.map { error =>
        <div class='row'>
          <div class="alert alert-error">@error.message</div>
        </div>
      }
      @input(registrationForm("name"), "Username")
      @input(registrationForm("password"), "Password", 'password)
      @input(registrationForm("confirm"), "Confirm Password", 'password)
      @input(registrationForm("realName"), "Real Name")
    </fieldset>
    <input type="submit" class="btn" value="Register"/>
  }
}

You can see from the source there that I've crafted my own helper function to keep things nice and DRY and avoid having to write out the twitter bootstrap required structure over and over.

Running and Wrap Up

Right about now the solution is done so with a simple play run you should be able to register and view current registrations. Excellent. If you want to skip everything and just get my code you can clone the Github Repo.

OK so it's not the most useful app in the world - that was never the intention. But look at the code - it's very terse and the time required to write this post was considerably more than the time required to implement the solution. Joy!

The Little Sacrifices

Sat Apr 28 23:00:00 UTC 2012

Late night ramblings...

Productivity can plateau very quickly - especially in a technical process. No matter how good you get at the process or approach, if you keep doing the same thing you can only get so good at it or do it so fast. It's possible, however, by sacrificing some of your precious time to go beyond that plateau. There are things we, the developers, do as part of our development process that are so small and insignificant that we do them without thinking

Clearing down the values in LocalStorage when debugging a web app
Copying assets from one folder to another (aka the macgyver deploy)
Navigating to http://localhost:8000 or 9000 or some other port
Typing rake generate preview into the terminal about 20 times an hour
These are but a few things I've had to consider today alone

Generally these things are so small we'd just do them by hand, right? But if you take the time to automate these processes or even just refine them in some way it can do wonders for you productivity going forward.

For example the clearing of LocalStorage in Chrome - spent a few minutes writing a bookmarklet that you can execute with 1 click. The macgyver deploy? Write a rake/make/jake/psake/cake task for it. Better yet use Guard or Watchr and perform the deploy on save! Navigatin to various different localhost ports? Try my alfred script to minimise the keystrokes. rake generate preview - alias that bad boy to rgp! Save yourself those extra keystrokes - you might need them later.

Listen - specifics aren't important here. You'll have your own small repetitive tasks and you'll be able to find your own ways to automate, improve or refine them - but this is the point. You probably don't. At least not as often as you should. Losing a second here and there on one task means very little in the small. In the large, across many of these tasks, well thats a different story. It's hard to justify spending 10 or 20 minutes on something to automate a task that takes 5 seconds to run but even if the time doesn't repay for a while then the fact you have less of a cognitive shift between your primary task and this secondary one should allow you stay in the highly effective, productive mindset to get the job done.

So give it a go, even as a thought experiment - think about the habitual development tasks you do and how you could automate them - you could well suprise yourself (like I have been recently).

Strapping Young Lad v0.1.2 Released

Thu Apr 19 23:00:00 UTC 2012

I have just released version 0.1.2 of Strapping Young Lad. Now we are getting to something that works a bit better across different platforms (now I just have to prove it to you). This release sees the inclusion of multiple token replacement. For example if I create a .ladconfig in the source repo with an array of tokens

token: ['__NAME__', '__NAMESPACE__', '__AUTHOR__']

The user will be prompted to specify the values for these tokens before the engine goes off and does the replacement. The __NAME__ token is a bit special - you still need to specify the project name as part of the command line call so if SYL come across this token in the the list of configured tokens it will allow you to default that value without having to type it again, e.g.

lad git@github.com:strapping-young-lad/Loam.git laddy

  Cloning git repository                OK
  Loading configuration                 OK
  Parsing tokens and stuff              
    name (laddy): 
    namespace: test.lad
    author: kouphax
                                        OK
  Processing files                      OK
  Processing directories                OK
  Moving project files                  OK

  Done processing 137 file(s)
  Done processing 22 directories(s)

Notice I didn't need to specify the name of the project again and then I was prompted for the second and third token values. Simple stuff.

Next up

So there are a few outstanding efforts to go in now. Namely push some more templates into the SYL repo and spin up a site that lets you search said templates and document all the little features of SYL. They will come in time.

This is the life!

Infoshare 2012 Talk

Thu Apr 19 23:00:00 UTC 2012

Slides

Resources

I mentioned a lot of technologies during my talk so here are the ones I mentioned

Web Frameworks

Design

Web Containers

Cross Platform

Server Platforms

Parse
Kinvey

Cloud Services

Testing & Debugging

Beta Testing

Comments

So how did it go? From my perspective it could have been better - I'm not sure why started "ummmmming" all the time - I had plenty to say (and I think I talked too fast as well). I just started in a semi-flustered state for reasons unknown. I seen a few decent comments on twitter so I guess it was fairly well received thats what matters. If anyone has any other observations, no matter how harsh, let me know.

One more note - Gdasnk is awesome - the technical people are so engaged and enthusiastic, the place itself is really interesting and I managed try my new favourite shot - Mad Dog. Thanks for the opportunity Gdasnk and Infoshare. Hopefully I can back here and spend more time in the city soon.

Na zdrowie

Strapping Young Lad - Project Template Token Replacer and Bootstrapper

Mon Apr 16 23:00:00 UTC 2012

Today I have released the super simple, rough-as-hell, barely MVP Strapping Young Lad(Source). It's fulfils a need I keep having but never really fixing.

Evelvator Pitch

SYL, or lad for short, will take a git repo, clone it and replace a bunch of tokens in it with the values you've supplied.

Say Wha?

This means you can create a bunch of base solution/project types for any language and SYL will simply do some basic file/folder renaming and file content replacing for a given token. For example doing the following,

lad /path/to/git/repo MyNewProject

Will do the following things,

Clone the repo at path/to/git/repo into a temporary folder
Remove the .git folder from the repo
Check for the existence of a .ladconfig in the freshly cloned repo

If one exists it loads the config setting
Otherwise it falls back to the defaults

Replace all instances of the configured token (eg. __NAME__) in all files, directories and file contents with the project name (in our case MyNewProject)
Copy the new folder over to your current working directory (./MyNewProject)

What this allows you to do is have a standard project template for pretty much anything (.NET, Ruby, Scala - listen I mean anything - if it a collection of folders and files in a git repo then you set) and create a new instance with a few keystrokes - saving you time and potential RSI.

SYL was inspired by WarmuP which is itself a gem but it requires you to be running .NET which for a lot of my requirement isn't going to cut it.

If you want to see what a typical template project looks like try Amir's Loam repo. This is a WarmuP project but works just fine. With SYL you can create a .ladconfig file to override some of the conventions - get the source README linked above.

Whats Next

Yep it's pretty damn basic right now but it satifies most of my needs right now. There are a few things I want to add for myself but I didn't release this for ME, I am pretty sure this will be of use to someone in some community in some shape or form. With that in mind here are some features I think would take this to the next level,

Specifiying multiple tokens e.g. __NAME__ and __NAMESPACE__ and __YEAR__ etc. etc.
Post processing actions (e.g. bundle install or npm install or sbt compile or something like that)
A catalog - I have a github organisation with no repos setup right now but this would be a brilliant place to hold a load of templates compatible with SYL, don't ya think? Then I could roll a site like rubygems or npmjs.org and make the projects searchable etc. If you build it, they will come. Maybe, who knows.

So if you like it - HIGHFIVE - if not - it's all good. Now go break it and criticise me.

So we'll criticise him, because he can take it.... [The Dark Knight, Approx.]

Scala - A Tragedy in Two Parts - Part 1

Thu Apr 12 23:00:00 UTC 2012

The tragedy being we don't use it all the time!

This presentation is the first part in the 2 part series "Scala - A Tragedy in Two Parts". In this presentation I go fast and dive into a range of Scala code snippets pointing out some of the weird and wonderful features of the language.

Recorded Session

Scala - A Tragedy in Two Parts - Part 1 - The Language from James Hughes on Vimeo.

Slides

Scala - A Tragedy in Two Parts

Thu Apr 12 00:00:00 UTC 2012

handsfread - Hands Free Page Scrolling Bookmarklet

Tue Apr 10 00:00:00 UTC 2012

Alfred Hack for Web Developers

Mon Apr 02 23:00:00 UTC 2012

I've been a fan of Alfred for some time and use it to make pretty much everything a keystroke away. I've thrown a very small little extension that reduces the time to jump between console and newly started web app. Generally during development we need to run our app jump to localhost:PORT and make sure it's working. If, like me, you have a lot of different apps and platforms (Scala, Jekyll, Rails, Sinatra, Express to name a few) then PORT becomes the one thing you always need to type and the browsers auto predict is generally not that helpful in this situation.

Alfred to the rescue here, just create a script as shown below and then running alfred and typing l 9000 you can launch your default browser at http://localhost:9000. Very simple but saves me heaps of time at the minute.

Rails is not a gateway drug

Mon Apr 02 23:00:00 UTC 2012

WARNING!!! - a negative opinion on Rails lies ahead. Everything here needs to be taken with an IMHO and, if you feel so inclined, tell me to go mind my own business.

I know I'm stepping into very dangerous territory here but I've come to the very solid conclusion that Rails, Ruby's heavily opinionated web framework, is not the gateway drug people claim. I've heard the claims that if you give Rails a try, coming from some other web stack on any other language, you'll fall in love with Ruby and Rails and write a big long blog post about how you are leaving .NET (or Java or whatever) behind. I really dont get it.... the only feeling that Rails leaves me with - having been using it in anger for about a month now - is frustration. But before you drive rusty nails into my hands and feet let me at least explain where I am coming from.

Previous attempts

I've been binging on Ruby for the last few months but this hasn't been my first time - it's been my only successful time but certianly not my first. A year previous I had given Ruby/Rails a shot and it never stuck (in fact not once but at least twice). Coming from a predominantly .NET/ASP.NET MVC background I decided the best approach would be to cut my teeth on a medium sized web based project and therefore jumped right into Rails. It only took a short time before the whole effort felt like a slog full of weird semi-magic behaviour. I gave up.

Doing things differently

This time around I decided to do things a little differently - rather than run with a web project I just tinkered with the language and eventually tried porting a .NET project over to Ruby. Fair bit of Ruby code and a small sinatra app later - it clicked. I got the whole Ruby love affair, it was lovely and I was three lines into my "F**k you .NET, you're worse than Java and I hate you and you smell" blog post (not factual!!) when I realised I needed to give Ruby more time to bed in and, out of fairness, I had to at least experience Rails.

Present Day

And with that I discovered that Rails had been my problem. I find it incredibly frustrating. I don't want to dive into bashing Rails, thats not my intent, - I'm sure people find it really productive - but it's not for me. So here is a very high level executive summary of the problems I've had with Rails

The Command Line Interface

The CLI stuff has just a little bit too many features and options that make it a difficult thing to approach without extensive research. When I watched some of the Railscasts I found myself going "wait, HOW did he do that?" more than once. Choice is good but surely Rails is supposed to be easy for the beginner? Yet I feel the default options are not the ones I would have picked. You know what - I want to be hand-held once in a while, I'm proud to admit that.

Too Much Magic

Rails is very heavily convention based but often there is simply too much convention and I've been left baffled a few times. Some of the convention is well documented and some of it less so. When a convention is not obvious it becomes magic and it really feels Rails has a lot of "magic". Thats not something I want. To be honest I'd rather write the same 2/3 lines of boilerplate if it meant that, to a new developer, they could understand the convention better. Take this very trivial example,

class HomeController < ApplicationController
  def index
  end
end

And the same in ASP.NET MVC

public class HomeController: Controller
{
	public ActionResult Index()
	{
		return View();
	}
}

To me that one line return View() tell me that this action will return a View, and now I can infer the convention of home/index.cshtml that wee bit easier. When you start adding in more conventions, a little bit of extra code makes all the difference. Conventions don't scale unless clearly marked IMHO.

Bloatyness

When you generate scaffold in a Rails app, as a typical new developer would, you end up with a awful lot of files. If I generate a new entity for example this is what I am greeted with,

--> rails generate scaffold MyEntity name:String age:Int                         !6580
      invoke  active_record
      create    db/migrate/20120403130542_create_my_entities.rb
      create    app/models/my_entity.rb
      invoke    test_unit
      create      test/unit/my_entity_test.rb
      create      test/fixtures/my_entities.yml
       route  resources :my_entities
      invoke  scaffold_controller
      create    app/controllers/my_entities_controller.rb
      invoke    slim
      create      app/views/my_entities
      create      app/views/my_entities/index.html.slim
      create      app/views/my_entities/edit.html.slim
      create      app/views/my_entities/show.html.slim
      create      app/views/my_entities/new.html.slim
      create      app/views/my_entities/_form.html.slim
      invoke    test_unit
      create      test/functional/my_entities_controller_test.rb
      invoke    helper
      create      app/helpers/my_entities_helper.rb
      invoke      test_unit
      create        test/unit/helpers/my_entities_helper_test.rb
      invoke  assets
      invoke    coffee
      create      app/assets/javascripts/my_entities.js.coffee
      invoke    scss
      create      app/assets/stylesheets/my_entities.css.scss
      invoke  scss
      create    app/assets/stylesheets/scaffolds.css.scss

Don't get me wrong - it's nice to be given all these things for free, but it's ultimatley stuff I end up ripping out anyway. In .NET there is a running joke that the "Empty ASP.NET MVC Project" Template is anything but. To me this is the same issue except it's something that I'd have to clean up everytime I use the generate scaffold command. This seems worse to me.

Current Thoughts

I'm not sure were Rails is supposed to stand. I hear it's great for beginners or for big sites but I couldn't recommend it for either right now. I realise I should really give it more time before rejecting it entirely but I feel that would be more time wasted. Rails doesn't make me more productive, at least not right now... perhaps once I repeat the same steps over and over and they become natural I'll see a greater boost in productivity (Ruby on the other hand felt natural and great right away). Perhaps if I shun the generators and roll my own stuff then I'll be more productive - but then why not use another framework that comes without all the gubbins I don't need? Sinatra is nice and simple and I can build upon it rather than breakling it apart. Ramaze seems like a great "next step up" from Sinatra but without as much (perceived) bloat as Rails. I've also followed the recent Kickstarter debacle from Yehuda - and I agree with him and his mission.

So what I am trying to say is - if you are a person wanting to give the Ruby world a fair shake - DONT start on Rails. Start simpler and later, if you want, when you've been throughly sold on the Ruby way, try Rails to see what you think. Start with Ruby, port some code, write a Sinatra app, watch the Railscasts as they are amazing then consider immersing yourself in Rails.

Educate Me

Finally - educate me. I'm not some sort of oracle proclaiming the death of a framework that thousands of people love. I'm not going to make anyone stop and think "I've been doing wrong all my life - screw Rails". If I'm wrong - tell me. If you think I need to give Rails another look - tell me why, tell me how. If you care - show me what I'm missing.

Government Digital Service

Mon Apr 02 00:00:00 UTC 2012

Setting the Selected Option in a Select with Selenium in Scala

Sun Apr 01 23:00:00 UTC 2012

Almost a perfect bit of alliteration there in that title - right? Anyway I've been doing a bit of work with bare-metal Selenium stuff in Scala recently and one of the things that kept me stumped for a while was setting the selected option for a particular select element.

I looked around the interwebs and didn't really find anything - now I'm not saying it isn't out there, I am certain it is, but I couldn't find it. So I decided to document how I did it in case any lone sole is stuck in a similar situation...

  driver.findElement(By.id("year")).
    findElements(By.tagName("option")).
    get(2).
    setSelected()

So here I am simply finding the second option of the "year" select element and setting it selected. I know it looks obvious now but trust me I was scratching my head for an age on this one.

I hope it helps some one, eventually :)

Behaviour Driven (Development|Design) in .NET

Wed Mar 28 00:00:00 UTC 2012

Done Spoked at DevWeek

Tue Mar 27 23:00:00 UTC 2012

So I made it through my talk at DevWeek which might sound a little odd but it took a whole bottle of cough syrup to do it. Not having been sick for most of the winter my body decided a day before DevWeek to go into shut down. I spent Tuesday in bed with very little voice. Thankfully the head colds hold on my vocal chords relented this morning and I think I managed to project as best I could.

So initial thoughts of how it went? I honestly think I'm getting better at this presenting thing so that must be a good thing, unless I was TERRIBLE before. The 2 main annoyances that have been raised thus far have been

1. Too Short

I did my usual "talk really fast and finish earlier than planned" thing I sometimes do. But there was plenty of discussion afterwards which I tought was interesting. FWIW when I present this to myself it takes longer. My internal monologue must be slower than the way I speak. Apologies to anyone that felt it was "too short".

2. Not enough Visual Studio

I opted to play it safe end embed code into my slides rather than code live - I didn't go unnoticed and I had a few comments that "moar codez" would have been good. Totally agree.

Other Stuff

I opted to change my slide deck between submitting them for print and presenting. I did it out of desire to devlier a better, more structured, talk (my first set being quite short as I was going for a "free-form" type of presentation). Probably not the most professional approach to do this but at least I can say my heart was in the right place :) Here are offending articles BTW

Code that was presented is available, as always, in/on my Github Repo. Thanks to all who attended and any other feedback you have throw it my way so I can get better at this talking malarky.

Also thanks to the folks at DevWeek for the opportunity - hopefully you'll have me back at some point.

Slim Generator for Octopress

Thu Mar 22 00:00:00 UTC 2012

So I've been overhauling my web site, which is based on Octopress, lately and have had to jump into the working of Octopress/Jekyll. It's obviously not the most complicated system in the world but hey it was new to me.

Anyway my current setup features a lot of html based pages (the default markdown approach didn't fit due to styling needs) and it was getting rather noisy. So I wrote, as there didn't seem to be one out there, a Slim generator. And here it is, step back it's VERY COMPLEX....

module Jekyll
  require 'slim'
  class SlimConverter < Converter
    safe true
    priority :low

    def matches(ext)
      ext =~ /slim/i
    end

    def output_ext(ext)
      ".html"
    end

    def convert(content)
      begin
        Slim::Template.new { content }.render
      rescue StandardError => e
        puts "!!! SLIM Error: " + e.message
      end
    end
  end
end

Just drop this file (call it anything) into your plugins folder in Jekyll/Octopress and any file with the slim extension will use this converter to generate HTML. No more noisy HTML for me!

rbenv and CodeRunner

Thu Mar 22 00:00:00 UTC 2012

You know CodeRunner right? If not maybe you should go check it out. It's and aptly named little tool for running code snippets in various languages much like what LinqPad does for the .NET folks. I've been using it extensivley recently to test out little Ruby and Scala snippets. Thing is though - I hit a snag. By default CodeRunner will run the system ruby executable (1.8 in OSX) and I generally use rbenv along with version 1.9.whatever so some of my scripts started behaving a little... odd.

But hey it's an easy fix,

Open your CodeRunner Preferences
Select the Languages tab
Select Ruby in the Language List
Change the run command from ruby $filename to ~/.rbenv/shims/ruby $filename

Before

![](/images/blog/coderunnerruby/before.png)

After

![](/images/blog/coderunnerruby/after.png)

Blam - global rbenv version is now the goto version for CodeRunner. OK so it won't work with rbenv locals but I am sure you can tweak it to make it work (I haven't needed it so never looked).

Tinkering with Spark: A Micro Web Framework for Java

Tue Mar 20 00:00:00 UTC 2012

You can't call yourself a language these days without having your own Sinatra clone. Not even, it seems, if you are Java. I know - it sounds odd that Java, the language that turns XML into stacktraces, has a micro web framework of it's own - Spark. I'm well aware as of JRE 1.6 has a built in HttpServer class but there is such as thing as too close to the metal :) Also Sparks logo is fairly epic.

Spark's site has a great little piece about the "why" and that is what stuck with me. Especially this bit,

Its intention is to be used by Java developers that want or are required to develop in pure Java.

I know a few people who are "required" to develop in pure Java and anything that might make their lives easier would be welcomed. Being completely out of loop with Java since before Maven made an appearance I had a lot of catching up to do just to get the project running. In fact I went with SBT as it made things a lot easier for me and my mavenless brain. I did have to tinker with the dependencies in the ivy2 directory due to a known issue but if I can do it anyone interested in this can to - if not give me a shout.

App.java

A Spark app simply uses main as an entry point. Within the method you can call certain methods to set up routes and filters and whatnots that your app will respond to. For example

import static spark.Spark.*;
import spark.*;
import java.io.*;

public class App {

  public static void main(String[] args)  {

Standard Java stuff here, bringing in some of the necessary spark resources,

    before(new Filter("/") {
      @Override
      public void handle(Request request, Response response) {
        boolean authenticated = request.queryParams("password").equals("secret");
        if(!authenticated){
          halt(403, "Incorrect password, hacker alert, hacker alert!!!!");
        }
      }
    });

Spark allows us to define filters that run before and after requests. It is also possbile to restrict them to certain routes or simply catch all routes - for example my filter only matches the "/" route. A couple of other things are worth mentioning here,

The Request object gives you access to many things you would expect i.e. the query parameters. It gives you so much more than this but I'll leave the docs to describe this.
Just like in Sinatra it is possible to simply halt the request chain at any time. Here I am halting execution unless the request conforms to my very strict and highly secure authentication scheme.

    get(new Route("/") {
      @Override
      public Object handle(Request request, Response response) {

        // extract the name from the request
        String name = request.queryParams("name");

        // set the response type
        response.type("text/html");

        // return some html
        return "" +
          "<DOCTYPE html>" +
          "<html>" +
          "  <head>" +
          "  </head>" +
          "  <body>" +
          "    <h1>Hello " + name + ", Spark here.  Howrya?</h1>" +
          "  </body>" +
          "</html>";
      }
    });

  }
}

Finally I am defining a route ("/") that responds to the GET verb and introspects the query parameters returning a very complex view. Routes can also be parameterized (/user/:name for example). Even given the slightly verbose nature of Java syntax this example seems clean enough (albeit a rather simple example).

SBT Build File

For the curious among you (you got this far so you must be slightly curious... and rather odd) the SBT file required to get this bad boy running is nice and simple,

resolvers += "Spark Repository" at "http://www.sparkjava.com/nexus/content/repositories/spark/"

libraryDependencies ++= Seq(
  "spark" % "spark"  % "0.9.9.3-SNAPSHOT" exclude("org.apache.commons", "commons-io")
)

A quick sbt run command and we are running a standalone Spark app (powered by an embedded Jetty server). Aces.

Mopping Up

OK so thats the basics of the framework but then again there really isn't much other stuff besides picking a port and redirecting. This means there is some things still missing that would really make this framework really stand out,

View rendering. Even if it is just plain HTML it would be a big leap forward (turns out an issue is already open too).
Static Content. Would be nice to define a rule for serving static content without having to define a route or filter manually
Session support. Again an issue and pull request is already available for this.

Good work perwendel keep it up :)

As always my code samples are available on Github.

Speaking at NDC Oslo

Mon Mar 19 00:00:00 UTC 2012

Mike Cohn, Dan North, Robert C. Martin, Udi Dahan, Gojko Adzic, Billy Hollis, Laurent Bugnion, Venkat Subramaniam, Brad Wilson, Phil Haack, Rob Conery, Greg Young, Zach Holman..... and then me. So someone was drinking when deciding the line up for NDC Oslo right?

Seriously though I've been given one of the best opportunites of my career to talk alongside some of the shining lights of our industry at NDC Oslo so I better not drop the ball. Thank you to whoever gave me the opportunity. But hey if I mess up I've got another 2 shots at it... thats right I'm speaking not once, not twice but thrice.

Still blown away by this, if you can't tell - given the calibre of people talking this is going to be a highlight of the year for sure. Also it couldn't have come at a better time - there is so much I've wanted to say about things like the OSS world in .NET, Mono and mobile development. Now I've got the perfect opportunity.

Now whats this .NET thing I keep hearing about :)

sentimentalizer: A Ruby port of Sentan

Mon Mar 12 00:00:00 UTC 2012

In a bid to be at least 1% less stupid I decided to jump into Ruby. Properly this time, no tinkering with Rails or watching a few videos. So the first thing I did was take @martinrue's Sentan project, fork it, made it run on mono (just because), ported it over to Ruby and renamed it sentimentalizer (but keeping all necessary attribution to Martins work in the README of course). I did this for two reasons,

I wanted something fairly meaty to actually port over to Ruby in an effort to get a more-than-superficial feel for the language
I might have a need for such a service in an upcoming hackathon (though the original .NET version would have been fine too)

This is in no way some sort of new project or intended for some production environment.

So what is it? sentamentalizer is a quick and dirty sentiment analysis tool. With it you can loosley class statements as either positive or negative (and get a breakdown of how that classification was derived). You "train" it with datasets for both negative and positive control statements and then feed it sentences you want analysed. At the minute the engine sits behind a super super super, did I say super already?, simple sinatra app that you submit statements to and it returns a simple JSON response with the sentiment result (negative, positive, neutral) and probability. If you want to train the bitch you've got to change the api.rb file or update the files in the data\positive and data\negative folders.

Plans? Yeah I might poke around some more, make into more of a standalone learning service or something (again another feature I might need soon anyway). But for now - feel free to review my shitey code Rubyists (I assume there are better ways to do things, standard appraoches that I'm not using etc.) and offer suggestions.

Once again thanks to @martinrue for doing all the hard-work on Sentan

Tinkering with minion.js

Thu Mar 08 00:00:00 UTC 2012

Yay a technical post...

minion.js is a micro framework providing classical inheritance strategy for JavaScript plus many other goodies. It can be used in both browsers and node.js. Now I don't want to get caught up in any of this classical inheritance for JavaScript is an anti-pattern stuff so just take this framework for what it is - a decent way to define both client and server side models for you JavaScripts!

I've already said it has a host of goodies that can be useful for creating a nice flexible set of models and associated helpers. This includes,

Class and inheritance support
Modularity
A cross model pub/sub mechanism
Statics, Singletons, static methods and properties
Dependency management
Some other stuff I've yet to touch upon...

The code will more or less speak for itself here so lets dive in (the full suite of tests and sample node.js/express implementation can be found on my GitHub thingy).

Classes

One of the first things I did was try out the whole inheritence thing. I create a Fruit base class and provided a Strawberry implementation on top of it.

minion.define("fruit", {
  Fruit: minion.extend("minion.Class", {

    isInSeason: function(){
      return this.inSeason;
    }
  })
});

minion.define("fruit", {
  Strawberry: minion.extend("fruit.Fruit", {
    inSeason: true
  })
});

Using the Strawberry class is pretty damn simple. First thing you need to do is a one off simple configuration of minion,

minion.configure({
  classPath : "js"
});

This just tells minions dependency manager where to start looking for your class implementations. We can then start using the Strawberry class easily using minions require method

minion.require("fruit.Strawberry", function(Strawberry){
  expect(Strawberry).to.not.be(null);
});

Static Objects

Static objects are pre-initialised Singletons in minion.js and that makes a lot of sense. I created a Grocer static by simply extending another minion type Static.

minion.define("fruit", {
  Grocer: minion.extend("minion.Static", {

    init: function(){
      this.stock = 0;
      this.subscribe("purchase", this.buy);
    },

    sell: function(){
      this.stock--
    },

    buy: function(){
      this.stock++
    },

    checkLevels: function(){
      return this.stock;
    }
  })
});

Pretty unsuprising but minion will new up a shared instance of this class the first time it is called. You'll see a little hint of the pub/sub mechanism in the init method as well so lets jump on that.

Pub/Sub

So minion has a nice little pub/sub mechanism built right into the classes as well. So lets say a customer buys stuff from the grocer by publishing his/her request - sure beats just taking it without asking!!!

minion.define("fruit", {
  Customer: minion.extend("minion.Class", {
    buy: function(){
      this.publish("purchase")
    }
  })
});

Minion once again handles all the internal wiring up of this stuff and it just works. Nice.

The Rest

There is more to minion.js, I've just given you a very high level view but you should dive a bit deeper by looking at the official docs. There is a few bits of handy sugar sprinkled on the minion types (proxy, scoped setTimeout and setInterval). The pub/sub also goes a bit further provinding Notifications that allow a sort of async callback/response mechanism which can be handy.

Now I've never been a fan of these sorts of Class frameworks - I've never really seen the need but I must admit minion appeared at about the right time. I've been playing with a nice way of structuring mongoose Schemas/Classes and I think minion here might be a nice fit.

Anyway check it out. Yay a technical blog with more to come!

minion.js

Sample Code

Taking the Stairs

Tue Mar 06 00:00:00 UTC 2012

Ya'll about ready for another silly analogy? I sure am....

I work on the 6th floor and recently decided to start taking the stairs. Mostly because I've let myself go and I really need to drop a few pounds and those stairs are a bit of a hard slog. But it's worth it. It's funny how much of a better understanding of the building layout I get by just putting in the extra effort and going up the stairs. The stairwell itself is pretty bland but I see extra doors, and I now know were many of doors could lead.

So the analogy... Taking the stairs is a bit like the whole software craftsmanship thing. Sure I could just take the prescribed route, the default stack so to speak, and as with the stairs I'd probably get where I was going for a lot of the time. But you know what? I'd not know much more than the average person. What if that lift was over capacity, out of order or just smelling of urine - I'd be able to manage just fine without getting stranded on a random floor wondering what to do next, if that door I never tried went were I wanted. Imagine the stress involved if I was already late for a meeting. Crickey!

So I put the extra effort in to understand things, and it's already given me a better perspective on things. It's shown me other options, the ability to decide if the what I am doing is a good approach or perhaps there is a quicker, better, healthier route.

How does that grab you?

I really need to get off this analogy trip I'm on FFS!

The Estimation 4x4

Sun Feb 12 00:00:00 UTC 2012

This post is dedicated to one of the most influential managers I've ever had the pleasure to work with.

I'm of the opinion that the moment you start trying to apply complex mathematical rigour and time to estimation you've entered into a broken system. You're wasting time trying to protect yourself from finger pointing managers or getting a kicking from old school "us or them" lawyer types. None of that adds anything to the project except a big stick to beat people with and make sure they work themselves to death based a a random stab in the dark that is somehow made more realistic by the application if science. Ultimately though if the initial guesswork is wrong, and it's seldom on the nose, the science means sweet FA. So why bother? You're pleasing the wrong people. But then those people hold a lot of power. So as long as we can keep them happy with the minimum of effort life should be sweet.

The Answer

Ladies and gentlemen I bring you the Estimation 4x4 - guaranteed to knock those super science estimation techniques out of the park.

The Science Bit

Got a project to estimate? I've done your homework for you - 4 people, 4 weeks. BOOM! Mind Blown! Now get on with delivering that project buddy. Oh wait it's a big ass project - 4 people, 4 months. Oh wait you're limited by deadlines and/or resourcing - adjust as needed (3 people, 5 weeks or 5 people, 3 weeks).

Balancing Act

Careful now, 9 women dont make a baby in 1 month so don't assume that scaling one side up to cater for the other will get the same results. Theres a balance that needs addressed. This is real science after all - repsect the rule and the rule will respect you.

You Gotta Be Kidding Me

Dont believe me? Try it out next time. Do your pseudo-scientific enterprise-ready finger in the air analysis. But hey see when that project is finished see how close the 4x4 would have brought you to the real result. I bet 9 times out of 10 it was as accurate as your real estimate, except you could have started much sooner....

#winning

Caveat Abstractor

Thu Feb 09 00:00:00 UTC 2012

Ramblings from 30,000ft. The word abstraction is starting to sound odd.

Every time you create an abstraction there is a very good chance you've introduced unnecessary complexity into a solution. It should never be done alone - it should be a team decision. The goal of an abstraction is to simplify but in its untamed natural form an abstraction will only add complexity.

As programmers it's easy to fall into the trap whereby we want the code that we write to be the last time anyone ever has to write that code again. We dream of creating reusable works of art that ever other developer will use and never be concerned with the low level implementation ever again.

This is wrong and impossible. An anti-pattern even!

It baffles me that anyone would create an abstraction that they didn't have a need for at the time. It's wasteful and degrades the readability, grokability and all those other good "ilities" we use. Of course I say this like i don't do it myself but I'd be lying to you and myself.

An abstraction, by definition, is an intentional hiding of implementation and knowledge but surely understanding of these things will lead to a higher chance of success in a project? The person that created the abstraction has that understanding but it's like they dont want anyone else to know how it works. They're ready to take that secret to their grave.

When you start to abstract you need to remind yourself you are building a specific solution and not a framework (unless you are of course). Abstractions should be done only within the scope of the project and you should have a strong case to back up its creation (make the decision a team effort). This may sound like overhead but if you find it difficult to justify the creation odds are the abstraction is unnecessary.

Abstractions should not be grand, they should be light and convey their intent clearly by other people (pair programming and peer review will help weed out any verbose abstractions).

If you are creating an abstraction for anything other than DRY, odds are youre probably creating it for YAGNI. You'll start considering irrelevant use cases and having to cater for insane edge cases. This will happen even if you are taking a strong test driven approach - you'll concern youself with the workings of the abstraction rather then the problem you are solving and begin creating endless "what if" tests to cater for all those "0.00000001% chance of happening" edge cases. When this happens youre going to end up in mediocrity. Mediocre abstractions attract more abstraction. Even in a shared codebase bad abstractions create a certain amount of implied ownership to the person that created it. Inevitably someone will come along and create an abstraction around your abstraction (even if they have access to the original source). This in turn will likely become YAMA (Yet Another Mediocre Abstraction) attracting yet more abstraction.... You can guess where I am going with that one... Its abstractions all the way down and everytime you abstract you've decreased system comphension and likely destroyed another future developers soul.

In the same way guns dont kill people, people do - Abstraction dont kill projects, developers do. You've only got yourself to blame (been there done that, will be again). A nice simple abstraction, at the right time (e.g. When needed) can be expressive and helpful but too often we put the abstraction before the need.

So consider abstractions as "guilty until proven innocent" not the other way around. Solve YOUR problem - not every other problem, not until you need to.

On Craftsmanship and Understanding, I Guess

Tue Jan 31 00:00:00 UTC 2012

You can stand at the station and wait for the train. You know the one. It's the one that's always late, always packed with people, head down staring at the ground. Or, you can walk the tracks with your own two feet. When you walk the tracks you take in your surroundings, you understand the route and you see the alternatives. You wont beat the train everytime but when you walk the tracks you'll have better stories to tell when you get there.

Enterprise Scale

Thu Jan 26 00:00:00 UTC 2012

Enterprise. What a broken, misued term. Its meant to sound big, huge even. I guess many years ago it used to mean big, huge, big business serious work.

But when you think about it, its none of those things really, not any more. Even the biggest multi-nationals entire enterprise backbone throughput is probably a small fraction of Lady Gagas Twitter feed. Enterprise is just that - one enterprise. Social is global, its all of us at anytime and that is so much bigger than some old enterprise. So if you try and pimp out your CV by using the term enterprise - the only people who are probably going to care are the managers and execs that think their little boring legacy system is unique and special... Do you really want to be sold that sort of job? All bytes are equal, some are more equal than others - Pah! Some are more mired in unnecessary abstraction, contracts and unread, out-of-date doucmentation but they are still bytes.

Using Parse in .NET and WP7

Thu Jan 05 00:00:00 UTC 2012

UPDATE: Being completely wrong in assuming WP7 supports dynamic which apparently it doesn't the current build wont run in WP7. So another item on the list is a version that doesn't rely on dynamic I guess. Sorry for misleading anyone on that front - I did say it was rough and mostly untested right?

Parse is a site aimed at providing a simple service for pushing and pulling data on mobile devices. You can create applications and store your mobile data there, create users, provide logon/off mechanisms and even achieve push notifications. There are 2 ways to access Parse

Via its natively compiled binaries (on iOS and Android)
Via a its Rest API (for all non iOS/Android devices and desktops)

Now I wanted to try and create bindings for the iOS library so I could use it for a Monotouch application I've been playing with but truth be told the whole process confused the hell out of me. Thing was the REST API gave me everything I needed so I started recently to write a dynamic wrapper around the API. Currently built on .NET in Visual Studio (Mono on my list) with a few sanity tests it works well enough to put it out there incase anyone was wanting to do something similar and could extract value from it.

It's pretty early at the minute - in fact all you can do with this first spike is basic object CRUD. No querying yet, no special user management features etc. But they will come. So what could you use it for? Well a number of things,

WP7 Cloud Storage
Monotouch (and Monodroid) storage until someone smarter than me creates proper bindings to the actual iOS frameworks
It's an object store so why not use it to store data on any web app?
Just for tinkering.

It makes use of dynamic from .NET 4 so you can kind of pass it arbitrary stuff and get arbitrary stuff back (Parse is schemaless so this makes sense for me). It also means you wont have to create concrete classes for every single thing you want to store.

If you want to run the tests you'll need to sign up for Parse, create an app and update the consts in describe_Parsely to point to your new app (and authenticate).

Sample Uses

// create your parsley API
Parsley parsley = new Parsley(applicationId, masterKey);

// create new class
dynamic response = parsley.Create("MyClass", new { CoolProperty = 1  });

// do other cool stuff
parsley.Update("MyClass", response.objectId, new { CoolProperty = 4});

dyanmic myObject = parsley.Retrieve("MyClass", response.objectId);

parsley.Delete(myObject.objectId);

Outstanding Tasks

Lots. But here is the main ones I see right now.

Compile on Mono and show MonoTouch some Parse love.
Samples (for WP7, Monotouch, possibly Monodroid)
Add Queries
Add User Management

It's probably fairly rough right now and could do with some TLC if anyone wants to pitch in? I've given it a name (Parsley) but that in no way implies this is going to be a long running project or anything. I just like naming things. Get it while it's hot, the code is in my usual place.

Pre-Post Update

Rather than update this blog post I decided to tack this on here. With BitBucket experiencing some issues at present I spent about 10 minutes hacking in the user management features. No tests written yet (oh no I have broken all the rules and must be beaten with oversized crayons) so lets just say that it's very edge, very experimental at the minute. Also bonus - returning a collection of objects is implemented as Query but again totally untested (fast and loose - it's how I roll son). There be dragons there therebe!

Happy hacking.

Can I have my WP7 MVP now please? Kidding

A Nancy Module that Behaves like a Rails Controller... Mother of God

Fri Dec 30 00:00:00 UTC 2011

If Nancy is inspired by Sinatra I thought I'd have a little fun a create an abstract class that created modules that kind of mimic the behaviour of Rails controllers. Code first, ask questions later...

public abstract class RailslikeControllerFor<TEntity> : NancyModule where TEntity : class
{
    public RailslikeControllerFor() : base("/" + typeof(TEntity).Name + "s")
    {
        Get[@"/"] = Index;
        Get[@"/new"] = New;
        Post[@"/"] = Create;
        Get[@"/(?<id>[\d])"] = Show;
        Get[@"/(?<id>[\d])/edit"] = Edit;
        Put[@"/(?<id>[\d])"] = Update;
        Delete[@"/(?<id>[\d])"] = Destroy;
    }

    protected virtual Nancy.Response Index(dynamic context) { return 404; }
    protected virtual Nancy.Response New(dynamic context) { return 404; }
    protected virtual Nancy.Response Create(dynamic context) { return 404; }
    protected virtual Nancy.Response Show(dynamic context) { return 404; }
    protected virtual Nancy.Response Edit(dynamic context) { return 404; }
    protected virtual Nancy.Response Update(dynamic context) { return 404; }
    protected virtual Nancy.Response Destroy(dynamic context) { return 404; }

    protected Nancy.Response AsView(object model = null)
    {
        string method = new StackTrace().GetFrame(1).GetMethod().Name;
        string entity = typeof(TEntity).Name;

        return View[string.Format("{0}/{1}", entity, method), model];
    }
}

To use it you just subclass the class and override the necessary methods. Like so.

public class QuestionModule : RailslikeControllerFor<Question>
{
    IQuestionsRepository questions;

    public QuestionModule(IQuestionsRepository questions)
        : base() { this.questions = questions; }

    protected override Response Index(dynamic context)
    {
        return AsView(questions.GetAll());
    }
}

What the base class does is create the necessary CRUDdy urls with a base route starting with the name of the entity you pass in, pluralised in the easiest way possible. In this case /questions. The rules match what you wold get for a controller in Rails.

Verb	Path	Action
GET	/questions	index	display a list of all questions
GET	/questions/new	new	return an HTML form for creating a new question
POST	/questions	create	create a new question
GET	/questions/:id	show	display a specific question
GET	/questions/:id/edit	edit	return an HTML form for editing a question
PUT	/questions/:id	update	update a specific question
DELETE	/questions/:id	destroy	delete a specific question

There is a teeny-tiny bit of sugar in the AsView method that will look up the corresponding view. So the module above would resolve its Index method to questions/index and the view engine would find the correct file (eg. for Razor - questions/index.cshtml).

I've already said it's a bit of fun but I have actually used this at least once - there is probably more that could be added but it'll do for a first spike at least. Any use to anyone?

Standing at the cusp of 2011... WTF?

Fri Dec 30 00:00:00 UTC 2011

I'm not sure where this post is going to go in terms of content. I'm thinking a bit of full disclosure might be quite cathartic but at the same time do you really want to know the ins and out of my somewhat awful 2011? Compromise - I'll keep it short. Possibly.

2011 (The Bad Parts)

2011 was a pretty terrible year if I'm honest. It was the year my wife suffered two miscarriages, I got into my first car accident, stuff kept breaking and I totally dropped the ball in work when I burnt out (I'm certain I'm still feeling the effects of that). Oh and yes on top of all that I was diagnosed with severe depression for which I am now fully medicated. But you know what? I'm not alone - I know a ton of people for whom (look at me I said whom - how posh) 2011 was grim.

So lets cast aside all the personal stuff, we've all had our problems, and let me talk about the burn out. Long and short of it was that I took too much work on and eventually delivered unacceptable results for them all. I have something akin to ADD when it comes to work and unless I am suitably challenged or stimulated my productivity drops to pretty much nil. 2011 was full of time consuming mundane boring work for me and as I tried desperately to pull personal value from the work (or hobbies) I started making mistakes, over committing and generally under achieving. Lesson learned - I need to learn to say no and sometimes I just need to suck up the fact that we can't all be doing the best part of our jobs ALL THE TIME. Sometimes we need to just knuckle down and get on with it. Just not for a full year...

Anyway I became somewhat disillusioned with my employer and, truthfully, Northern Ireland. I like other people to be excited about the stuff I preach but I felt there was a lot of disinterest and empty words. This annoyed me and I guess I just started pushing harder, which I guess made me look quite angry and pushy. This obviously wouldn't help. When I looked outside of my company - Northern Ireland didn't seem that much better. So I felt a bit out of place and I had 2 choices ahead of me.

Up sticks and go somewhere that I felt would be more receptive to my extreme nerdyness and desire to share it.
Stick around, my company and Northern Ireland and try and make a difference.

I chose the second, not being to be hero or anything but for more pragmatic reasons,

My wife got pregnant again (due soon yay!), the economy is terrible and it would be too much of a risk right now
If I stayed in Northern Ireland, or even Ireland, I don't believe there is any other company that would support me (aka let me get away with) the things I do. I'm not the most predictable of sorts. I work weird hours and do strange things. Kainos have always been there for me and supported me. Having done interviews with people from various levels from various other companies the state of other IT firms makes me seethe with anger.

It is better to invest in people and risk them leaving than to not invest in them and risk them staying

So I start 2012 with a big challenge set for myself. That sounds about right. The more I think about Kainos, my employer, deserves more credit than I give it. We have been in a space thats been all about big business types with big enterprisey solutions and delivered well. It's not an area that exactly screams enthusiasm and I think it will take a while to draw people out of that kind of insular thinking. Kainos certainly seems to be better than most in this area. Now enough gushing about Kainos (who are hiring BTW)

2011 (The Good Parts)

OK OK 2011 wasn't all sob stories and dullness some really cool things happened.

My CodeSlice initiative in work (meet at least once a month, talk nerdy things, get Kainos to pay for free pizza) didn't die. In fact in the hands of one of my colleagues, Jonny Holmes, it flourished. Attendence was never HUGE but it was consistent and varied. May long this tradition continue and hopefully we can expand it..
I organised and participated in not one but two hackathons (internally). Once again Kainos covered the costs for our weekend long hack sessions. Turnout was decent (about 8-12 people) with all types turning up - managers, placement students, technical architects etc. The experience was enough for me - which is a clever way of saying the end products weren't as good as envisaged. But we drew so many valuable lessons that the second one was infinitely more successful that the first - live and learn! And, oh lord, the plans I have for the first half of 2012 around hackathons are going to be legendary!
I talked, and not just to myself but to real people, real people who were not paid to be there! I talked, outside of work, a total of three times. I know the number isn't huge but it was late on in the year before I decided to get out there. Already topped that number for 2012 and the year hasn't started. Double figures would be good next year thank you.
I attended GOTO Amsterdam were I got to meet a lot of people I have deemed awesome (because they are). Being in rooms with like minded people (who are generally much smarter than me) saying things that make you realise you are not insane for thinking things could be better really strengthens your beliefs!
Shared more. All the stuff above I did because I wanted to get my thoughts out there - I wanted to be educated as well as educate. I was happy to look like a fool if it meant I didn't look like a fool later. Thats hard for an introvert but damn it I gave it my best shot. I know people, the one who only know me through my writing on Yammer or half-read blog post, think I am a purist (I'm not - perfect is the enemy of better), or angry (not me, frustrated at times but never angry), or opinionated (openly opinionated perhaps) or even annoyed that everyone isn't dancing the happy geek dance like myself (as long as I'm not dancing alone on the dance floor like a sleazy old man stalking girls half his age I'm happy). Thats fine. People reckon I risk detracting from the message I am trying to communicate. Fair enough. Perhaps I need to tone it down sometimes - but I can't see that happening - it's no in my nature. I think it's better that people see me as a purist and take some of my ideas away rather than being half-assed and them not taking anything on board! It's inevitable that when I try and challenge peoples conceptions of technical things, especially in the open, it's going to leave a sour taste in some peoples mouths. But thats how I roll and thats how it should be. Sorry but there is more on the way. But I take as good as I give so come at me bros!
Won a years to subscription to Tekpub on Christmas Day thanks Rob 'Santa Clause' Conrey.
Put my foot in it a few times. Yeah sounds odd I am classing this as a good thing but Scott Hanselman gave me permission, and I quote

Chris Sells told me once, If you're not getting in trouble with your boss at least twice a year, you're likely not pushing the envelope hard enough. Two slaps a year might be the cost for 10 successes. If you're not moving forward, well, you're not moving forward.

2012 (More like 20AWESOME! AMIRITE!)

So enough dwelling in the past - how am I going to make 2012 be the awesomest? Well thats easy - set some real fecking goals. I failed to do that last year (outside of my typical career goals during my annual reviews of course) and ended up coasting in parts. Not next year - hells no. I got me a bucket list (stuff to do before the year dies - not me - unless that comes first - oh god I never thought of that). TO THE LIST!

2012 Bucket List (In No Particular Order)

Release a screencast
Release a game
Appalachian Trail
GiveCamp NI
Talk/Present as often as possible
Learn a new language
Contribute more to OSS

So let me explain some of these, as I am sure they mean sweet FA to some of you.

Release a Screencast

A lot of my blog posts last year were a kind of "Introduction to Technology X" and I think that sort of knowledge can be better shared via video and real coding. Coupled with my desire to go a bit deeper in my technical posts but always getting distracted with other blog posts I think 2012 is the year I try my hand at producing a real life proper screencast (or perhaps a series). With decent production values and what not. The topic? Well I am currently thinking Nancy but I need to really get my head around it a lot than I already do - but thats fine. Time will tell on that one.

Release a Game

Game programming is an area I've failed to venture into and I feel it is about time. I wan't to do something really simple, something mobiley possibly using ImpactJS because it interests me. I'd like to release said game into an app store as well - just to watch it get flamed and to say "I've done it"

Appalachian Trail

This is a codename for a current unfunded project of mine. The goal is to get a more fuller understanding of the spectrum of mobile development approaches, platforms and technologies. So simply put generate a bunch of super simple (though still useful) app ideas and using every conceivable type of technology from native to web and all in-between deliver decent quality apps across all the major platforms through hackathons, team challenges, school and university outreach, app camps etc..

Sounds awesome doesn't it?

GiveCamp NI

GiveCamp was founded by Microsoft Developer Evangelist, Chris Koenig, in Dallas, Texas in 2007. His vision was to bring together passionate developers and local charitable organisations for an entire weekend, and see what would happen.
In late 2012 GiveCamp is coming to Northern Ireland.

Thats what the official site says (I should know I put it there). So late next year we will run GiveCamp for the first time in Northern Ireland. A weekend long charity focused public hackathon. It's going to be a belter!

Talk/Present as often as possible

I enjoy talking about nerdy things. I got a taste for it last year when I went public (having been talking internally to Kainos staff for years) and in 2012 I really want to get out there as much as possible. I find it an amazing way for me to learn as well as understand whats going on in the industry. So far I am speaking in Dublin in January, Devweek in London and a few other un-announced venues.

BTW if anyone can recommend conferences I should pitch for, or want me to come over and talk about anything in the .NET/JavaScript/Mobie world(s) (see my blog for the kind of thing I know about) - let me know!

Learn a new language

Everyone says this one right? Yeah so I spent last year looking at node.js and Ruby and I think it's time to get out of my safe - I'm thinking F# - yeah that sounds sweet!

Contribute more to OSS

I head-dived into OSS .NET last year and loved it. Many people see .NET as a very restricted, Microsoft led platform were every modern web-app is MVC3 with Entity Framework and WCF rules the web service world. Thats total bull. .NET has a very vibrant, very active OSS community. There is almost no need to use any MS technology ever. But lets not forget MS have really started to embrace and support OSS in a big way. Lets hope that trend continues!

In Summary

Boooo to 2011, Yay to 2012. See you around the same time next year when I have another failful year and promising myself a better one in 2013 :-P

Code Coverage and the Forces of Good

Tue Dec 27 00:00:00 UTC 2011

Code coverage often does more harm than good. It removes the focus from testing the expected behaviour of the code to just writing tests to make sure all your code is hit (gamifiying the process to its own detriment). People will bend tests, often in a way that no typical use case could, just to make them run as they want. This is unrealistic and paints a false picture of the behaviour of your code.

That being said - code coverage is a tool and it shouldn't be blamed for enabling bad habits (a bad workman always blames hi... well, you know the rest). So can code coverage be applied in a less obvious way? Indeed it can.

Business users invariably end up using things like Google Analytics, Gauges or Flurry to discover the most valuable areas of the solution and tailor accordingly. Makes perfect sense, right? So why not use code coverage, during development, as a mechanism to identify redundant areas of code, or discover feature hotspots that could benefit from refactoring or performance tweaking? Why not indeed?

As any system evolves there will be behaviours that simply don't need to exist to satisfy the overall behaviour of the system. Some code and it's related tests will still hang around. It happens and with the right tools you'll be able to see those areas - MightyMoose will show you hit counts for lines of code for example. So by ensuring tests are reviewed as part of your code review process (or refactored during pairing) you'll also get reporting of redundant code or feature hotspots for free.

So if your project is already reporting on code coverage thanks to some great architectural mandate why not at least use it for something more positive?

Ugh. He's on about something mobiley again. Lord save us all!

Tue Dec 13 00:00:00 UTC 2011

Good timing right? Fresh of the wave of my "lets have a natter about mobile" blog posts HP went and did something good (for a change, right?). After the hash that was made of the TouchPad HP went and open sourced the webOS software. Bundled with this software is Enyo a mobile framework for WebKit browsers. Enyo was to be one of the main development platforms for webOS - JavaScript on the front end with node.js services for the backend. The future was bright. Then the TouchPad flopped. Ack well.

On the other hand Enyo is a very nice little framework and in the hands of the Open Source community could be a real winner. I've had a chance to tinker with over the last few days and thought I'd help get the word out by outlining how to write the obligatory by law ToDo app. It is currently somewhat similar to the official FeedReader tutorial but it could be easily extended to add more features. I recommend you look at the original tutorial for more info about how the project is structured as there are certain conventions that may seem odd to your typical web framework.

The entire application, which looks like this (yeah yeah it's not the most beautiful thing on the planet - I suck at design).

Was knocked together with the following bit of code

enyo.kind({
  name: "TodoApp",
  kind: enyo.VFlexBox,
  components: [
      {kind: "PageHeader",  components: [
        {kind: "VFlexBox",  flex: 1, align: "center", components: [ {content: "Todos"} ]},
        {kind: "Button", caption: "Clear Complete", onclick: "clearComplete"}
      ]},
      {kind: "RowGroup",  components: [
        {name: "newTask", kind: "Input", hint: "Enter new ToDo here..."},
        {kind: "Button", caption: "Add", onclick: "addTask"}
      ]},
      {kind: "Scroller",  flex: 1,  components: [
        {name: "list", kind: "VirtualRepeater", onSetupRow: "getTask", components: [
          {kind: "Item", layoutKind: "HFlexLayout", align:"center", components: [
            {name: "taskStatus", kind: "CheckBox", style: "margin-right:10px;", onChange: "completeTask" },
            {name: "taskDescription" }
          ]}
        ]}
      ]}
  ],

  create: function(){
    this.todos = [];
    this.inherited(arguments);
  },

  addTask: function(){
    var task = this.$.newTask.getValue();
    this.todos.push({ description: task, done: false });
    this.$.list.render();
  },

  getTask: function(sender, idx) {
    var todo = this.todos[idx];
    if (todo) {
        this.$.taskDescription.setContent(todo.description);
        return true;
    }
  },

  completeTask: function(sender, evt){
    this.todos[this.$.list.fetchRowIndex()].done = sender.getChecked();
  },

  clearComplete: function(){
    var buffer = [];
    for(var i = 0; i < this.todos.length; i++) if(!this.todos[i].done){
      buffer.push(this.todos[i]);
    }
    this.todos = buffer;
    this.$.list.render();
  }

});

Those of you with some experience in Sencha Touchwill notice that the declarative syntax is quite similar to STs except slightly simpler. So lets break out some of the features that brought this example together.

{kind: "Button", caption: "Clear Complete", onclick: "clearComplete"}

This is one of the component declarations that make up the suite of components that form the "kind" (Enyo speak for component or object). This is somewhat similar to the DOM (COM anyone? Component Object Model) except in JSON form. You can specify a component type (kind) and apply properties (which may or may not do something depending on the kind). Events are also declared inline (see onclick) and map to function names within the main kind declaration. Obviously if this declaration started getting too big and bloated I could decompose it into smaller components all handling their own stuff.

One other thing not highlighted in my demo is that backend service calls can also be defined within this component model and then called programatically. While this felt alien to me initially it is pretty similar to the way inline stores are declared in Sencha - except slightly less engineered.

{kind: "Scroller",  flex: 1,  components: [
    {name: "list", kind: "VirtualRepeater", onSetupRow: "getTask", components: [
      {kind: "Item", layoutKind: "HFlexLayout", align:"center", components: [
        {name: "taskStatus", kind: "CheckBox", style: "margin-right:10px;", onChange: "completeTask" },
        {name: "taskDescription" }
      ]}
    ]}
]}

The scroller definition above has a few nice features. The use of "flex" to manage different screen sizes or viewport resizes and create very nice flexible layout. The VirtualRepeater which is a templated control that generates items from a collection based on the defined template. The Virtual part is useful for large lists and ensures that only the minimum amount of items are rendered (just before they are needed).

create: function(){
    this.todos = [];
    this.inherited(arguments);
}

Create is essentially our component constructor and allows us to set some things up in advance. Notice the use of the inherited() method - this ensures the superclasses create method is also called (inheritence chaining)

this.$.list.render();

this.$ is a handy property. It is essentially a hash of all the named components that are direct and indirect children of the defined component. This makes it very easy to access a certain component if you want to manipulate it.

getTask: function(sender, idx) {
    var todo = this.todos[idx];
    if (todo) {
        this.$.taskDescription.setContent(todo.description);
        return true;
    }
}

getTask is the method responsible for populating an Item template in the repeater for each row. This is where I bind the values of the current collection to the Item kind.

There you have it - yet another ToDo app spat out into the world! Just what we needed :)

It not all Unicorns and Rainbows

Yeah Enyo isn't perfect (shock horror). There are some quite annoying and some serious bugs that I have noticed in my hacking.

Data Association

When rendering the Repeater I can't see a way to easily attach data items to that item instance. I worked it out eventually by using this.$.list.fetchRowIndex() but I think you would need to then manage your own data association. Not a big issue but it didn't feel right to me at the time.

iPad/iPhone issues

The one big issue I had with the iPad was that text fields often didn't bring the keyboard up (more often than not). Obviously this could be a bit of a deal breaker but there could be a simple fix. Some other minor quirks included scrolling issues when multiple scroll areas existed and choppy animations in the kitchen sink style demo (Sampler).

The example apps display and work on the iPhone but they are super tiny. I am not sure if this is just a lack of correct formatting within my HTML or if Enyo is purely designed for larger screens.

Events

At one point I wanted to bind to an event of a component programatically (rather than declaring it inline). I still can;t find a way to do this and I wonder if this could be a bit of hindrence when you get into a truly dynamic application. Thought this could be me being a bit stupid as usual.

Conclusion

Enyo v1.0 is still young in the hands of the OSS community but it's started out strong and could well be a strong competitor to something like Sencha Touch if it keeps up this pace. Hopefully the community embraces it and allows it to grow. It could well be great.

So yes, so far I like it but I've only been hacking a few days. Hopefully I get to spend more time with it over the next few weeks.

The sample project is available on my BitBucket Repo if you want to tinker.

So This Mobile Stuff Then... (Techologies)

Mon Dec 05 00:00:00 UTC 2011

Do you have your mobile strategy yet? Do you have your "one true way" to cater for every mobile application need? If not, the world will suddenly end and you'll never get anywhere in the mobile market. Seriously. Well not seriously. Obviously that's a pile of sheep.

To be blunt the best "mobile strategy" is the obvious one. Do whatever suits the project best. Yeah, yeah your company or your presales guys may not like that, no prepackaged waffle to fill document with or spew out in front of customers. The easy route is seldom the best route. Seriously though if your playing it safe with web just in case that iOS project you've gotten involved in goes Android you'll be compromising all the way.

Just In Case

Ahhhh good old "just in case". Nothing produces more mediocrity through compromise than "just in case". It's up to you to use your head when it comes to deciding what is the best fit for the project but as long as all the stakeholders are aware of the constraints those choices put on the project then those "what if" scenarios can be dealt with later. We need to stop treating "mobile" as if it was a technology decision. Web vs Native is not the same as saying Entity Framework vs NHibernate. It's so much bigger than that.

Weighing up the options

So you got a killer mobile project and you want it to be awesome - just like all your projects. Before we pull out the "Native, Hybrid or Web" question we need to know what is out there. What tools, technologies or languages can we use to make the best solution we can?

Going Native

The problem with the native moniker is that people assume it means a different code base per mobile target and therefore crazy expensive, difficult to handle change, and very time consuming but that isn't always the case. It is entirely possible to produce truly native apps without having to have an expansive code base in every language possible.

Appcelerator Titanium

Appcelerator's Titanium is a good example of a platform that provides an abstract API via a common meta language (in it's case JavaScript) that allows you to build directly to native apps for each target platform. You write your code and send it off to the cloud for compilation (AFAIK there currently isn't an in-house option for building but happy to be corrected). So you've got a middleman that you have no control over. What happens if your app gets rejected due to some low level issue in the compiled code? What if the cloud service goes down when you really really gotta build? What if Appcelerator goes bust? I dunno maybe that's just mostly FUD but these are the questions that customers ask and they are difficult ones to answer. I realise I've lambasted the use of too much "what if"s in decision making but I've come up against these ones and they directly affect the current solution rather than only the future scope. They need addressing and that's difficult.

Having said the a brilliant example of Appcelerators usefulness is Wunderlist - available on a ton of mobile and desktop platforms and built with Titanium the whole project shares, AFAIK, a common code base.

MonoTouch, Monodroid etc.

MonoTouch and its ilk take a slightly different approach to the meta-language. MonoTouch, for example, isn't, as many believe, a cross-platform tool for building native apps. No, MonoTouch simply compiles to native code through direct bindings to Objective-C code. You are simply using the C# and the powerful features of the Mono/.NET Base Class Library. In fact it's even possible to bind to third party Objective-C code like Flurry, Three20 etc. (and many are made available by other developers).

The ability to share a common codebase comes from the fact that the same platform is available in MonoDroid (C# and Android) and Silverlight (Windows Phone 7). So sharing comes down to architecture. You can share code provided your code doesn't touch and platform specific code. You can create your own abstraction layer and work it into your build process or you can use MonoMobile.Extensions which gives you a common API already.

So in one way the MonoTouch stuff is simply the same as writing Objective-C but simply doing it using C#. This is nice for a number of reasons.

Mono has some very nice features like LINQ and Parallel programming.
Plenty of nice compatible frameworks - sqlite-net, catnap, restsharp and plenty more.
Syntax that is less noisy and easier to read

There are of course issues. Assuming most people who use MonoTouch will come from a .NET background and want to use those skills - when you start you'll realise just how nice Visual Studio is. MonoDevelop isn't terrible, it's just fine, but it's simply not on par with Visual Studio (would you expect it to be?). With the way MonoTouch works your are kind of stuck with MonoDevelop too. Your average VS2010 developer might easily get frustrated with MonoDevelop. An aesthetic concern but one that can put people off easily. That said XCode is equally as infuriating at times (bugs and quirks abound!).

It's not all roses though. What if Xamarin, the commercial sponsor of MonoTouch, pull the plug? What if Apple decide, as they tried to before, to ban the use of any intermediary language? These are questions that people will ask but again the immediate and real advantages always trump theoretical risks. One other thing worth thinking about - If you came from a Ruby background for example I'm not sure why you would use MonoTouch over Obj-C. Will this niche have an affect on Mono* in the long term? Who knows.

Mixing It Up aka (cough)_Hybrid(cough)_

So maybe you're a web guru/ninja/hero/wizard type person and you want to make use of those powers/skills in your mobile development but you want to get market visibility by creating a presence in the app stores. Perhaps you want to have some sort of reuse across platforms but aren't a .NET dev and Titanium isn't appealing. There a number of options in this area already.

PhoneGap/Callback

PhoneGap is easily the most technology in the hybrid space. It is exactly what you expect. A fullscreen UIWebView in which resources embedded in the application package are executed. These resources are given access to native device features such as device access etc. via a JavaScript API that hooks into native function calls. There are wrappers for numerous platforms (iOS, Android, Bada, Blackberry, WP7 etc.) but your code calls the same adaptive API. Things start to veer off course when you need to introduce custom plugins as these are native and so you need to target all your platforms. Not much PhoneGap can do in that case of course but worth pointing out.

With PhoneGap it is up to you to create the experience. Even if you want a simple straight forward native look and feel you'll need to simulate it. Results may vary depending on how good you are at that sort of thing. Of course something like Sencha Touch could be used to fake native but again what is native on one platform is weirdly alien on another and if you are targeting one platform only why not just man-up and learn that platform in the long term? At least give yourself that option.

NimbleKit

NimbleKit is in some ways a bit like MonoTouch meets PhoneGap . You can use it as you would PhoneGap as a transparent API the bridges native and Web Technologies but you can also create real life native components such a Navigation Controllers and Tabs. Currently iOS is the only supported platform (more support for iPhone than iPad) though an Android version does exist in beta.

The interesting thing about NimbleKit is that it allows you to make use of a range of technologies to create a really native application while at the same time making use of HTML5/CSS3 features to create a more compelling experience that becomes very difficult in native platforms (see my previous charting example in my last post).

Plugins are extremely simple as well. Just register the Obj-C class and call it from JavaScript. Very nice, very simple. This allows third party extensions to be bridged very easily too (Flurry for example).

The slightly annoying thing about NimbleKit is that it is currently very quiet. There is still activity but it's certainly not on par with the likes of PhoneGap. There are also some issues around stability. It's not falling over all the time but there are somethings that need to be done in a certain order that aren't properly documented. There is some minor speculation that Sencha have acquired NimbleKit and I'd welcome that with open arms because I like the philosophy around NimbleKit and it could do with some money behind it.

Options for Using Web Technologies

Be it hybrid or simply a mobile targeted web site/app there are a ton of options out there. An absolute TON!

Native look and feel, ultra rich frameworks
Truly cross-platform progressively enhancing frameworks
Frameworks for creating structure in your apps like MVC, MVVM etc.
Frameworks for offering responsive designs

Mind boggling choices that change and evolve on a near hourly basis. Here are just a few

Native Look and Feel - Sencha Touch

Sencha Touch targets high end iOS and Android devices and creates native-esque UIs (WebKit only). It's can be patchy on Android at times but the results you can get on iOS are very appealing. Version 2.0 is seeing massive improvements in performance across the compatible platforms. People who have experience in ExtJS will really feel at home here - other people will struggle with the short but steep learning curve (as with all Sencha frameworks).

Bear in mind that Sencha Touch is a framework not a product so there will be work involved in getting it to behave exactly how you want but the available architecture options such as the MVC approach and the Data Stores make it very powerful.

Cross Platform - jQuery Mobile

jQuery Mobile lives at the opposite end of the mobile framework spectrum. It takes the philosophy of jQuery UI (Progressive Enhancement, accessibility, themeability etc.) and put it into a mobile framework. So you construct a purely functional purely HTML application or site and with data annotations you let jQuery Mobile make the site as rich and functional as it can. This means provided your mobile device can render HTML the site or app will at least be functional. Butt ugly but functional.

This carries a penalty though. Even when used on the top of the line device there are some trade offs that need to be made. The experience is not as polished as it could be. It is possible to tweak it yourself and cater to the high end devices only but there is extra work involved. Again the result is still very nice and generally smooth but it is easy to nitpick little quirks in the navigation or styling.

MVC Frameworks - Backbone, Spine(Mobile), Sammy etc.

If you want to roll your own UI entirely but still need to create a scalable and proven foundation Backbone and Spine are two very interesting projects. It's worth noting that Sencha Touch has a powerful MVC architecture built into to and it is somewhat similar to these frameworks.

These sort of frameworks offer base classes for your typical project structure - Controllers, Models and Views. They provide the wiring between these abstract classes allowing the developers to focus on the important stuff. The controllers allow you to specify custom routing in your single page apps, the models give you structured persistence and sometimes relationships and the views give you a nicer lifecycle around page components and custom event handling.

It's often said these frameworks take some time to "get" - some people can use them with ease while others just think they introduce unnecessary complexity. I think they come into their own in the mobile app space but I do still struggle with the concepts at times.

Responsive Design - Skeleton

There are grid and UI toolkit frameworks out there that are aimed at creating a responsive design. Got a large screen? Make use of the better horizontal space and stack elements horizontally. On a small mobile device with tiny viewport? Stack the elements vertically. Skeleton is the perfect example of this using media queries to layout a grid system depending on available space. It also has a few nice widgets like Tabs and Buttons. Geared more around the web site rather than the app these frameworks are a handy addition.

Run out of things to say

I've covered a lot there but I wanted to get my thoughts out there in the world and hopefully educate and be educated. My tools of choice right now? Well RIGHT NOW I am tinkering with Obj-C and iOS as well as MonoTouch and did some NimbleKit hacking at the weekend. To me this sort of understanding is important and that's why I do it. You may argue that it makes you a jack-of-all-trades but I don't think so - this I merely trying to better understand the mobile ecosystem in the same way I try and grok the entire .NET ecosystem.

To me if you want to embrace mobile you need to understand it properly. If you're just sticking with a particular stack because it's familiar to you you are doing yourself and the project a dis-service. As many articles before me have said - you are not the programming language you code in - you need to embrace the polyglot nature of a good developer and do what is right for that project.

Hope you enjoyed this. That's me for December I think :). I have other things to do.

Rolling Your Own PhoneGap with MonoTouch

Mon Dec 05 00:00:00 UTC 2011

PhoneGap is a clever little idea. Take a transparent native wrapper for a mobile platform, stretch a 100% x 100% web view and provide a JavaScript API that acts as a bridge to native methods that control the device features such as cameras, GPS and Contacts. It brings all those lovely web people into the mobile space. Truth be told PhoneGap isn't powered by flaked unicorn and rainbow essence - it's fairly simple to achieve yourself.

So I was thinking, as I do when I am bored, that it would be super awesome if MonoTouch could do the PhoneGap. Turns out there have been some attempts in the past to create bindings but I barely understand the posts so I felt disappointed. Then curious. Then I started hacking. Then happy. Then bored again. The happy once more. Then slightly frustrated. Finally I started writing this post and I think I feel a bit tired because it's late an I've had a hard day.

Anyway what follows is a very primitive start to what is essentially a PhoneGap, possibly NimbleKit, for MonoTouch. Thats cool becuase not only would you be saying "Screw You Objective-C!!" by using MonoTouch but you'd be sort of saying "Hey MonoTouch - HTML5 just had your mum, twice" as well. Hey I'd use a NimbleKit/PhoneGap-like framework for MonoTouch if it was available and offered a clever way of mixing Native with Web but thats just me. Hence I am writing this post.

The Recipe

There are various components that would make up a framework like this.

A fullscreen Web View (lets call it the webside)
A controller that handles requests from the webside (lets call it the nativeside)
A JavaScript API on the webside that talks to the nativeside

Simple enough so lets start proving things.

The Webside

Using a simple Single View iPhone Project in MonoDevelop I added some local resources.

    |
    |_ www
    | |_ index.html
    |
    |_ AppDelegate.cs
    |_ MainViewController.cs

For now the index.html file has some basic content

<!DOCTYPE html>
<html>
    <head>
        <title></title>
    </head>
    <body>
        <h1>Hello UIWebView</h1>
    </body>
</html>

Next up I created a full screen UIWebView and asked it to point to the index.html I just created in wwww folder (which by the way should be set as "Content".

public override void ViewDidLoad ()
{
    base.ViewDidLoad ();

    using (UIWebView view = new UIWebView(new RectangleF(0f, 0f, 320f, 460f)))
    {
        // obtain path to actual content file
        string path = NSBundle.MainBundle.PathForResource("www/index", "html");

        // create an address and escape whitespace
        string address = string.Format("file:{0}", path).Replace(" ", "%20");

        // create url and request
        NSUrl url = new NSUrl(address);
        NSUrlRequest request = new NSUrlRequest(url);

        // load request and add to main ?view
        view.LoadRequest(request);
        this.View.AddSubview(view);
    }
}

Running this gives us what we would expect (I hope you expect this otherwise I'd be worried).

Webside more or less spiked for now. Next up the nativeside.

The Nativeside

So how is the native stuff supposed to respond to requests made via the mysterious JavaScript API from the webside. The approach that makes the most sense here is to set up an internal HttpListener that reacts to web requests. This means we can, thanks to the lack of a Same Origin Policy in UIWebViews via the file:// protocol, make Ajax requests to said listener to do stuff and return appropriate responses.

In the AppDelegate class I created a basic HttpListener that simply returns a simple JSON response of true if a request is made.

public HttpListener listener;

public void HandleRequest (IAsyncResult result)
{
    //Get the listener context
    HttpListenerContext context = listener.EndGetContext(result);

    //Start listening for the next request
    listener.BeginGetContext(new AsyncCallback(HandleRequest), listener);

    string response = "true";
    byte[] responseBytes = System.Text.Encoding.UTF8.GetBytes(response);

    context.Response.ContentType = "text/json";
    context.Response.StatusCode = (int)HttpStatusCode.OK;
    context.Response.ContentLength64 = responseBytes.Length;
    context.Response.OutputStream.Write(responseBytes, 0, responseBytes.Length);
    context.Response.OutputStream.Close();
}

public override bool FinishedLaunching (UIApplication app, NSDictionary options)
{
    window = new UIWindow (UIScreen.MainScreen.Bounds);

    viewController = new JsBridgeViewController ();
    window.RootViewController = viewController;
    window.MakeKeyAndVisible ();

    listener = new HttpListener();
    listener.Prefixes.Add("http://*:30001/");
    listener.Start();

    listener.BeginGetContext(new AsyncCallback(HandleRequest), listener);

    return true;
}

The Final Step - The Bridge

So we need to talk to this "server" from our JavaScript. Easy stuff - for now we just make a simple ajax call to our endpoint in the index.html (synchronous for code simplicity).

<script type="text/javascript">
    var request = new XMLHttpRequest();
    request.open('GET','http://127.0.0.1:30001/', false);
    request.send();

    if(request.status == 200){
        alert(JSON.parse(request.responseText));
    }
    else{
        alert("Error");
    }

</script>

Running this once again reveals that all is sweet...

Now What?

Well thats about as far as I've actually gotten but this is far enough to satisfy myself that it's possible to talk between the web view and the native stuff. Yeah I know it's not rocket science but I needed to prove this to myself. So whats next? Well I want see what I can do with regards to making real calls to real things, perhaps even venture into NimbleKit territory and generate native elements such as ViewControllers and Tabs - who knows!

(MYSTERIOUS EXIT MUSIC)

(FADE TO BLACK)

To Be Continued...

(EVIL CACKLE HEARD IN THE DISTANCE)

(END SCENE)

So this mobile stuff then... (Hybrid)

Sat Dec 03 00:00:00 UTC 2011

It's fair to say over the past year or so I've dabbled in a bit of mobile development. I've been involved in a number of projects (from a technical perspective),

Web Based, Tablet Focused Analytics Dashboard (Custom HTML and HighCharts)
Web Based, Tablet Focused Data Capture application (Sencha Touch)
Hybrid application for a teleco (jQuery Mobile and PhoneGap)
Mobile Hackathon (jQuery Mobile and PhoneGap and Parse)
Hybrid Application for a Bank (Sencha Touch and PhoneGap)
Various iPhone and iPad POCs for a health based project (MonoTouch, XCode)
Technology evaluations (Native Apple Development, Parse, Sencha Touch], PhoneGap, NimbleKit, jQuery Mobile, Backbone, Spine.js, Appcelerator, MonoTouch, Monodroid and plenty more...)
Consulted on various projects around the whole "native vs hybrid vs web" thing

That Hybrid Word

One thing I've found is that many clients are using the old "a web app will never be as polished as a native app" argument but TBH that really isn't the case. This is down to a lack of real understanding and abuse of the word in the industry. A bit of hybridism can go along way in creating a rich experience.

A Rose By Any Other Name...

Many people break mobile applications into 3 VERY distinct areas,

Native - app installed on the device, written using the default toolkit for that platform.
Hybrid - consists of a transparent "native" layer and a Web View into which HTML, JavaScript and CSS is loaded (generally stored within the native package). The nativelayer provides and API for the JavaScript model in the Web View to call. Think PhoneGap.
Web - HTML, CSS, JavaScript assets hosted on the web and accessed from a mobile device. Possibly some native-like features such as icons and removing the browser chrome.

The problem with the hybrid definition there is that it is, well, wrong. There isn't a clear distinction between the native and hybrid and there shouldn't be. If I spun up a Silverlight app that put a Web View in the middle to display some HTML I wouldn't call it hybrid. If I wrote an Android app that used a library written in Scala it wouldn't be hybrid. Or even if I wrote a Monotouch app using some bindings to the Flurry framework - nope not hybrid.

The Charting Example

A good example of "hybrid" is charting. There aren't a great deal of nice flexible charting libraries for iOS, but there is for JavaScript/HTML5. So some of our projects have made use of HighCharts and Sencha Charting inside a transparent UIWebView to render charts that are flexible and rich. I wouldn't bother pinning the hybrid tag to that app - it's simply making use of whatever technologies helped us get the job done in the best way possible.

I understand the use of hybrid in sales pitches or to simply convey meaning to people who are less technical but the problem is people are starting to develop opinions based on these words alone which is disatorous. I've had customers tell me that they want native because that hybrid stuff is ugly. True some hybrid apps are ugly as sin (Aer Lingus iPhone app for one) but then so are some native ones. It's unfair to collectively throw away and entire type of app because of poorly misunderstood words.

But, But, But... Reusability

People use the hybrid term to show that they have thought about portability. Write once, run anywhere. In that sense I guess using hybrid makes sense (but that sort of talk is really part of the sales talk right?). But then it would be just as easy to say that "elements of the solution will be written in a platform agnostic language to minimise code rewrites on different devices" - or something like that. In fact that would be better. That way you cover off more than "hybrid". You've included the things like MonoTouch or Appcelerator as well. You've moved away from implementation specifics at the point where they shouldn't matter anyway! Thats got to be a good thing.

Ah ha but what about Skills?

People go "hybrid" for another good reason - because they have no experience in the native platform. I used to be one of those people but I'm not convinced that this is a good reason on its own. If you want to make a compelling experience your limitation should never be your lack of skill. Get skilled. Obj-C isn't as scary as it seems - in fact I'd rather write Obj-C than Java. Dig deep - thats what we've got the internet for - understanding and learning (well that and porn).

Awareness and Education

So what am I saying? Well for a start I'm not saying we totally abandon the word hybrid and mock anyone who uses it. Hells no. It's still useful as I've already mentioned above. All I ask is that you make sure you know what you are talking about and that the people you are talking to know what you are talking about. That and don't just take the easy route because it's easier for you.

Chutzpah 1.3 Released

Sun Nov 27 00:00:00 UTC 2011

I missed the announcement last week the Chutzpah 1.3 has been released. This is a big release as it brings support for Jasmine - my BDD framework of choice for JavaScript.

Chutzpah is available as a plugin for Visual Studio but more importantly as a command line runner which allows us to integrate JavaScript testing into our continuious integration builds. Excellent stuff!

Out with the old...

Fri Nov 25 00:00:00 UTC 2011

... in with the new. So I've started rolling out my new site, expect somethings to be slightly broken as images and whatnot have been moved around. Spent a day or two designing a new site that was similar to the old one but slightly better. Eventually got frustrated at my inability to do any sort of graphical design and went the Octopress.

Still a few things to tweak, possibly add a splash of colour around the place and give it a bit more life. Also apologies to people who subscribe to my feed - looks like the new feed has pushed all the old posts out as new. But hey they are worth reading twice!

Speaking at DevWeek 2012

Thu Nov 17 00:00:00 UTC 2011

So it's been officially announced that I am talking at DevWeek 2012 in London (26th - 30th March). Looking at the names on the billing it's quite weird seeing my name thrown in there, perhaps some one made a mistake :-P

BDD and .NET
Behaviour Driven Design/Development is an interesting premise, but does it really work? What benefits does it offer and how can we successfully apply its principles in the .NET world? We can answer these questions by reviewing the principles around BDD and diving into the technologies that exist in the .NET world that support these principles.
From low-level unit focused testing technologies all the way up to the more abstract integration and acceptance level suite of specifications, automated testing and continuous integration, we will see that .NET isn't lacking in choice. As a bonus we will address the oft ignored testing of our JavaScript project components using tools such as Jasmine and QUnit.

The brief was written in haste and sounds a bit boilerplate but I will be giving it my all and really pull off an engaging session. I've been at the attendee end of a few bad sessions and I will do my damnedest not to put anyone through that kind of ordeal.

The line-up for DevWeek looks excellent and the pre/post conference workshops all look stellar - so much choice, I just need to find a way to clone myself.

If you're going - see you there, catch up with me and come throw tomatoes and/or abuse during my session. If you're not going, well sucks to be you then!

Wednesday Tip: C# Type aliases

Wed Nov 16 00:00:00 UTC 2011

Ever written code that looks quite like this?

Dictionary<string, KeyValuePair<string, string>> cache = new Dictionary<string, KeyValuePair<string,string>();

I have. Ugly and noisy isn't it! OK you could make it slightly less noisy by using var

var cache = new Dictionary<string, KeyValuePair<string,string>();

Or I suppose you could ahead and create yet another random type to abstract the implementation out of the way but thats just too much work for something internal to a class.

Turns out you can make use of alises. I've always known about using alises for namespaces to either resolve possible type conflicts or to make code a bit more understandable

using CoreWeb = System.Web
...
var x = new CoreWeb.Request()

So this works with namespaces AND types.

using Cache = Dictionary<string, KeyValuePair<string,string>();

This allows us to make the declaration above in a much nicer manner,

Cache cache = new Cache();

OK so there are way to many uses of the word cache there but you get the gist. Handy to know. Never knew this until today #youlearnsomethingneweveryday

Innovation Waffle

Wed Nov 09 00:00:00 UTC 2011

There has been a lot of internal chatter at my employer around innovation. A recent internal Hackathon produced some interesting, innovative stuff for using a Kinect in one of our projects and piqued peoples interest. Oddly enough my dad reminded me of a quote the other day.

"If I had asked my customers what they wanted they would have said a faster horse" - Henry Ford

Creating a faster horse is considered sustaining innovation - you're pushing boundaries but you aren't breaking into new markets. Instead, what Ford did was create the Ford Model T, the first affordable automobile. This disruptive innovation changed everything, breaking into new markets and created value where once no value existed.

To be truly innovative you need both a need and the technology to achieve it. The need doesn't even need to be particularly complex (in Fords case it was the need to go faster), the technology doesn't even need to be revolutionary - just applied in a different manner (in my employers case the Kinect).

But to really succeed in this innovation you need the ability to marry these with something. That something is a vision. Ford realised the need and he was capable of harnessing technology to address that need in such a way as to disrupt the market. This wouldn't have been possible without Fords vision. In fact he didn't even really need to understand the detailed nuances of what the public needed, all he needed to know was the basic premise. Arguably, even if he wasn't the engineer, he could have got someone else to manage the technology. Admittedly this opens up another avenue of discussion - the whole nature vs nurture aspect of innovation - but with a strong enough vision I'm not sure that matters as much.

Vision is the Glue of Innovation

It's great to have a need you want to address, it's even better to be able to apply all sorts of technology to that need but odds are it will all come to nought without a strong vision. Vision is the glue of innovation, while you need both need and technology they'll never really stay together without that binding vision.

GiveCamp Northern Ireland 2012

Thu Nov 03 00:00:00 UTC 2011

http://givecampni.org.uk/

Aint no backing out now son! In 2012 there is going to be a Northern Ireland based GiveCamp. The "Register Your Interest" placeholder site is now live at http://givecampni.org.uk/ so if everyone can start spreading the word and getting excited that would be super awesome.

Any tips, advice or help at this point is also really welcome.

Let the fun begin....

Nancy JSONP Hook

Tue Nov 01 00:00:00 UTC 2011

3 Nov 2011: As of Nancy v0.9 this is part of the core (same logic applies)

I needed to write a simple JSONP capable service recently and decided to run with Nancy. Turns out Nancy doesn't give us a way to do JSONP so I set about creating something that fitted my use case. The fruits of my very very short labour (thanks to the super-hyper-flexible-powerfulness of the Nancy Pipeline) can be seen here. The full source is available in the Github repo and hopefully we can get this pushed into the core of Nancy shortly (or at least part of the official repo one way or another) - just for the geek cred of course :)

Implementation Rundown

I took the same approach as the existing hooks (using the SassAndCoffee one as my implementation pattern) so created a static Hooks class with an Enable method that simply adds a new item to the end of the pipeline

/// <summary>
/// Enable JSONP support in the application
/// </summary>
/// <param name="pipeline">Application Pipeline to Hook into</param>
public static void Enable(IApplicationPipelines pipeline)
{
    pipeline.AfterRequest.AddItemToEndOfPipeline(PrepareJsonp);
}

The meat of the work lies in the PrepareJsonp method

/// <summary>
/// Transmogrify original response and apply JSONP Padding
/// </summary>
/// <param name="context">Current Nancy Context</param>
private static void PrepareJsonp(NancyContext context)
{
    bool isJson = context.Response.ContentType == "application/json";
    bool hasCallback = context.Request.Query["callback"].HasValue;

    if (isJson && hasCallback)
    {
        // grab original contents for running later
        Action<Stream> original = context.Response.Contents;
        string callback = context.Request.Query["callback"].Value;

        // set content type to application/javascript so browsers can handle it by default
        // http://stackoverflow.com/questions/111302/best-content-type-to-serve-jsonp
        context.Response.ContentType = "application/javascript";
        context.Response.Contents = stream =>
        {
            // disposing of stream is handled elsewhere
            StreamWriter writer = new StreamWriter(stream)
            {
                AutoFlush = true
            };

            writer.Write("{0}(", callback);
            original(stream);
            writer.Write(");");
        };
    }
}

Let me highlight a few of the key areas here.

Determine if Response need JSONP Padded

I decided to take a basic approach to determining if the request should return a JSONP response,

If the content type of the current response is application/json
If there is a query string parameter called callback

Here is what we need to satisfy that,

bool isJson = context.Response.ContentType == "application/json";
bool hasCallback = context.Request.Query["callback"].HasValue;

if (isJson && hasCallback)
{

I am sure this could be refined and open to suggestion (perhaps more applicable content types or extra possible configuration for the callback parameter). I wonder if it is a bit safer to restrict requests that are GET requests seeing as that is the only way to do JSONP? Hmmmm.

Next thing I set the content type to application/javascript as that makes perfect sense

context.Response.ContentType = "application/javascript";

Finally I wrap the original responses content in the JSONP "padding" (the callback function) and write everything out.

context.Response.Contents = stream =>
{
    // disposing of stream is handled elsewhere
    StreamWriter writer = new StreamWriter(stream)
    {
        AutoFlush = true
    };

    writer.Write("{0}(", callback);
    original(stream);
    writer.Write(");");
};

So all in all very simple but damn I'm proud that I got off my arse and contributed to such a stellar community. hopefully someone finds this useful.

Tinyweb.FluentSecurity v0.1 Released

Tue Oct 18 23:00:00 UTC 2011

Slight change here - FluentSecurity is now FluentAuth to avoid any confusion with a similarly named product of which there is some inspiration taken. Anyway on with the original post....

I have procrastinated long enough on this and it was blocking some of the other work I wanted to get on with. I have just pushed v0.1 to Nuget and launched the site sans documentation.

Not a great deal has changed since my last announcement, well it has but it hasn't been documented as it should be backwards compatible. Just a few more configuration options that I'll explain once I get the documentation rolled out.

Install-Package Tinyweb.FluentAuth

So go check it out and hopefully you'll find it useful - plenty of plans for expansion to other frameworks and some more API changes coming too. Now back to procrastinating.

refORM: Death to ORMs in .NET

Mon Oct 17 00:00:00 UTC 2011

CoffeeScript the Awesome

Mon Oct 10 00:00:00 UTC 2011

CoffeeScript the Awesome

Sun Oct 09 23:00:00 UTC 2011

CoffeeScript - "a little language that compiles into JavaScript" has been making a lot of noise in the web community with its inclusion in Ruby on Rail v3 and its swift rise in popularity on Github.

This session will look at the benefits this language brings over the native JavaScript alternative, the syntactical double rainbow it creates, the problems you'll face and of course how you can actually use it in different technology stacks such as Ruby, node.js, .NET and Java.

refORM: Death to ORMs in .NET

Sun Oct 09 23:00:00 UTC 2011

Data Access is a difficult area to master. There are plenty of frameworks such as Entity Framework, NHibernate, i/myBatis. These frameworks attempt to make data access simpler but while these products are widely used they can lead to leaky abstractions forcing us to work around constraints of the technology or introduce the nightmare of trying to figure out what is happening deep down in the frameworks internals.

In this session we will look at the various lightweight alternatives and the advantages they offer as well as provide rationale about why the more heavyweight approach isn't always the best approach.

Demo Code and Examples

DDDNorth Retrospective

Sat Oct 08 23:00:00 UTC 2011

So its the day after DDD North and I'm more or less getting packed and ready to do the long boring travel bit home, which is really less travelling and more waiting around. Anyway DDD North was pretty special. Excellent turn out, great location and above all some great talks. My talk (refORM: Death to ORMs in .NET) had a technical problem with the projector which meant the live demo section I wove in since last weeks talk had to be abandoned, shit happens. It also went very fast, the talk was over in about 35 minutes but at least there was plenty of banter and questions - so at least the people seemed to be engaged. Enough about my woes. I'll have the slides and the (super secret) demo code available as soon as get back home.

So what about the rest of the talks?

Commercial Software Development - Writing Software Is Easy, Not Going Bust is the Hard Bit - Liam Westley

Liam took us through some of his personal experience of working in the industry as a sole(ish) trader. He highlighted some of the things that you should really be doing to ensure you aren't "wasting time" and raised a very valid point - every time you take a support call you are bleeding money. Makes sense but I initially rebelled against this premise. I had wrongly jumped to the conclusion that if you don't get support calls you don't need a support contract - which in hindsight is pretty stupid of me.

He also highlighted that if the phone does ring, or a support issue is raised then you need to make sure you have all the information at hand. Detailed logging of crucial areas of your system, automated emails or tweets when the system detects something is wrong and generally making sure you don't need to rely on the end user to tell you what has went wrong. Again makes perfect sense. Obviously writing good code that is well tested prior to release is another winner in this area.

In terms of releases he re-iterated the old "release early" mantra. You don't want to waste time writing features that no one is going to use and you aren't really going to know this until users have the product in their hands - the minimum viable product.

All in all a great session, nothing massively new for me here but I love hearing this stuff from people with real experience in it - it really helps solidify my beliefs in good code and early release.

"The Happy Programmer" - Is It a Myth? - Andy Gibson

Looking the various aspects of a developers life that determine if it is truly possible to have a "happy developer". A fair amount of audience participation looking at the best office type, the rationale for having the best hardware and all that other stuff.

Continuous Delivery - Paul Stack

Certainly the highlight of the day for me. Many people I work with believe that CD is a long way off for many projects but there was so much hard evidence at this talk that debunked this theory. If it can be scripted or automated in anyway it should be. This includes HTML, JS, CSS, and C# code analysis, unit tests, acceptance tests, environment configuration, document generation, Sharepoint config and setup (yep that to), VM rollout, DB upgrades and backups. Everything really.

A great example of this was a massive search engine in Norway that turned their 2 1/2 year release cycle into once every 4 hours. This wasn't some big massive leap, it was done by gradually shortening the release cycle and tightening up the processes bit by bit till they got there.

A good point was raised on this topic later in the evening - CI/CD scripts should be the very first thing you do on a project. Then you build them up gradually as and when you need to. No point in waiting till the end of the first release do this - it'll hurt way too much and never get done.

The 10 Habits of Highly Effective Programmers - Dennis Doomen

The last session, and after being up since 4AM I was wrecked. But you know what? Even with the tiredness and the heat of the room I remained engaged. Again a lot of the topics in this talk I was already very familiar with but it was great to get a different perspective and new techniques for achieving them. Also managed to increase my "To Read" list with a few books from this session. Nice way to finish off the day.

Grok Talks

Through lunchtime there were a number of quick, informal grok talks. The one I want to mention was on "teaching your kids to code" and showed various tools/games like LightBot and SmallBasic - my son is still a tiny bit young for this sort of thing but he'll probably be forced to do some of this with me being his dad. I'd also have liked to have seen Scratch (from MIT) mentioned as this is a similar technology.

Overall the day was beyond my expectations - plenty of refreshments, goodies, decent food and great people. To be honest I really felt like a small fish presenting though - after seeing the other guys I felt my delivery was decidedly amateur. But hey thats why I am doing this - to get better. Looking forward to the next one! Thanks everyone.

Next up I have GOTOAmsterdam (this Wednesday). Funnily enough I need to find my passport ASAP as 2 minutes after jokingly saying "Wouldn't it be funny if I couldn't find my passport?" to my wife, well, you can probably guess what happened. So not only did I look like a dick in-front of my lovely wife I also managed to put my GOTO conf attendance in jeopardy - hell I really am a dick!

DDDBelfast Retrospective

Sun Oct 02 23:00:00 UTC 2011

So I'm no longer a conference speaker virgin. It was so good I did it twice in the same day. How'd it go? Went good I guess. From a conference stand point it seemed to fly by. There was a kind of laid back atmosphere about the whole thing and the numbers, while lower than expected were high enough to make all the talks feel comfortably occupied. I skipped the first block to get my talk prepared but managed to pull in a few talks after I got my first one over.

Other Peoples Talks

Behavioural Driven Development (BDD) with F# - Phillip Trelford

Big toss up here as I wanted to attend Andrea's Code Kata as well but this talk won out. It was a decent talk - a nice mix of theory and practise. Interesting to see Phils testing framework, TickSpec, in action, I must revisit this again some time soon (a promise I'll probably never keep :( ).

CQRS - Bringing Elegance to "Normal" Applications - Ashic Mahtab

The last CQRS talk I attended flew right over my head in terms of complexity. Since then I've done a bit of research and came prepared. I really didn't need to though. Ashic did a really good job of taking something that appears quite complex and conveying the message quite nicely. Interested to see his 250 LOC CQRS framework as well!

My Talks

refORM: Death to ORMs in .NET

Went quite well. I played it safe in terms of not going for any live coding but I think it would have been much better if I had (and plan to next time week at DDDNorth) as it was more dry than I would have liked. Lots of good discussion after the presentation that I might try and bring in next time too.

Coffeescript the Awesome

Slight technical problem resulted in the majority of the code samples being nearly unreadable for a lot of the presentation which was a total fail. I think I flew through the whole thing too - perhaps a bit of live coding would have improved this one too. That said the discussion lasted as long as the talk more or less which is good. Another problem was a number of people came in quite late and missed the first lot of slides which meant I was repeating the content at the end but hey thats fine.

Tinyweb Fluent Security

Sun Sep 25 23:00:00 UTC 2011

I've been working on a little project recently that is using Tinyweb. It's a typical little CRUDdy app and makes use of typical security features in a web app. Tinyweb itself doesn't provide any sort of custom security features out of the box but as it sits on top of ASP.NET you can avail of all the normal membership stuff. Problem is though - the default membership stuff, when working directly with it, is a bit cumbersome. I recently tinkered with a project for ASP.NET MVC called FluentSecurity that added a fluent syntax to configuring security for ASP.NET MVC applications. Taking a healthy dose of inspiration from this project I rolled a lightweight Tinyweb version of my own. I figure this might be useful to someone else so I have now created a new project specifically for this. It's probably totally broken in areas but the areas I've written tests around appear to work.

The source is available in a BitBucket repository so feel free to critique. Once I get it tested better, documents written up and generally applied a bit of sipt and polish I'll push it to Nuget and the fun can really begin.

Tinyweb.FluentSecurity

FluentSecurity comes as a 2 part solution

The Security class is the main entry point for configuring FluentSecurity and authenticating users
The SecurityFilter class is a Tinyweb filter that performs auth tests on the current request

Security --

The Security class provides a central location to configure FluentSecurity. Security is also responsible for executing the auth tests you pass to it.

Security.Configure(Action configurator)

Configure lets you define the configuration for you module. It accepts an Action that can be used to define rules for each handler. For example

Security.Configure(c =>
{
    c.For<RootHandler>().DenyAnonymousAccess();
    c.For<AdminHandler>().RequireRoles("Admin");
    c.For<UserHandler>().DenyRoles("Admin");
    c.For<SecretHandler>().AllowVerbs(Security.AllowedVerbs.GET | Security.AllowedVerbs.POST);
});

Tinyweb.Init();

We call Security.Configure just before we Init Tinyweb (though it can be done at anytime realistically speaking). The configuration block shows off most of the ways you can configure handlers e.g.

DenyAnonymousAccess will prevent, as the name suggests, all anonymous users from accessing this handler
RequireRoles specifes 1..N roles that are required to access this handler
DenyRoles specifies 1..N roles that aren't permitted to access this handler
AllowVerbs restricts the HTTP Verbs that can be used to access this handler (PUT, POST, GET etc.)
WithCustomRule (not listed) allows you to specify a function that can be used to apply a custom rule to each request

Each method returns the configuration object so it is possible to chain the calls to create more complex rules.

Security.Configure(c =>
    c.For<EditHandler>()
        .DenyAnonymousAccess()
        .RequireRoles("Author")
        .DenyRoles("Reader")
        .AllowVerbs(Security.AllowedVerbs.POST);

Obviously some combinations will not make sense, in fact AllowedVerbs needs to be tweaked as it may only apply to certain roles (so the combinations don't make sense right now). Currently FluentSecurity doesn't care and it probably never will if you try and create illogical combinations.

bool Security.Test(RequestContext req, HandlerData d)

This will run all pre-configured rules against the current request to determine if the request is authorised to continue. It will return a true/false result depengin on whether the rules pass or not.

bool granted = Security.Test(context, data);

IResult Security.Validate(RequestContext req, HandlerData d)

Implements the typical SecurityFilter use case. This function will call the Test method and returns Result.None() if the test passes (may be configurable in the future) otherwise it executes the Security.OnAccessDenied function and returns the IResult. If no OnAccessDenied is defined Result.None() will be returned.

IResult result = Security.Validate(context, data);

This is simply shorthand for now but may be expanded with some custom logic in the near future.

Security.OnAccessDenied

This property is used to define a function that can be used to return a result should access been denied. This is done lazily as it allows you access to the RequestContext and HandlerData so you could implement per-request handling of the Result (redirects or return URLs etc.)

Security.OnAccessDenied = (c, d) => Result.Redirect<AccessDeniedHandler>();

SecurityFilter --

The SecurityFilter class is simply a Tinyweb filter that performs the request validation for authentication. Nothing special here move along now.

For Now and Up Next

So thats all there is for now. Code is, as I've said, available at the BitBucket repository. I'm starting to build up some docs and stuff now and maybe tweak and add useful features. If there is anything you would like to see raise it as an issue and I'll get around to it.

Personal Systems of (Dis)Organisation

Tue Sep 20 23:00:00 UTC 2011

From the outside people probably regard me as a typically chaotic person. My desk is a mess, I'm not mad keen on the whole shaving thing, my writing is terrible and I am generally not where people expect me to be at any particular time. But there is, more often than not, a workflow that works for me hidden deep down in the chaos. Most people stick to one or two way to organise themselves (Todo Lists, pen and paper, iPad etc.). Not me. I'm still trying to find some sort of process that fits well with me but I'm starting to think that perhaps that actually is my process. I move from different mediums and tools quite a lot. Doing this isn't always easy given some tools just natrually try and lock you in. My current workflow involves a few tools,

Pen and paper (Work only). Nothing beats its it immediate and short term loggin of free form information. I have strict rules around it though.
Blank paper - no lines or binder holes or headers. I guess I subconciouslly view this a restriction perhaps?
Black pen (with a bit of red). I've totally gone off blue pen - it makes my notes look messy (they are though). Pencil has been overtaken by the black ink pen these days though I am not sure why
Wunderlist. List management. Cloud syncing. Free. I moved from Things.app when I realised they just don't seem to listen to their customers. Cloud synching is essential. Wunderlist offers me a cross platform and web based solution for doing everything I need quickly. I use this for medium-longer term "listable" tasks such as,
Project Deliverables
General day to day work tasks (though not shopping lists - see later)
Things to blog about
Wish lists for presentas and books and things (shared with family who also do the same thing)
iOS 5's Reminders app. Yeah ANOTHER list manager. This one is for very short time lists such as shopping lists. I find polluting Wunderlist with this sort of list very annoying. I don't use it that often but it's the best app I find for ultra fast entry of lists.
Trello. The latest addition to my supposed system. It's essentially a multi-level list system. Kind of like a cross between a Kanban board and a Todo list application. I am currently using this for more and more things. Essentially higher level list management that makes more sense to have a pseudo workflow or a board to collect realted lists and collaborate with others. The key to this one is everything becomes more collaborative but it's much less restictive than something like AgileZen.
Books (To Read, Reading, Finished)
Hackathon Voting (Ideas, Timings, Dates, Objectives etc). Allows a bunch of people to vote and discuss relevant things for the next hackathon. We also use checklists for things we need to prepare in advance etc.
House renovations (one list per room). Shared between my wife and myself.
AgileZen. Amazing site for Kanban and Scrum. Essentially a Kanban board with enough flexbility/features that means we can tailor the workflow to suit the needs of the project and team. I use this for work projects where feedback on times and bottlenecks is essential. Can't say enough great things about this.
Whiteboard - Ultra transient information. I use this during real time brain thinking sessions. An old friend remareked that the process of getting out of your chair and picking up a pen changes the way you think and creates a more open environment for ideas. Generally items on the whiteboard last a couple of minutes. Anything important can be photographed and sent to Evernote or email. I do my best work at a whiteboard. If I am blanking I get up and start doodling - inspiration eventually comes. Currently there is a picture of Silos (pronounced Sigh Loss) - The Evil Data Overlord - makes sense to me.
Evernote. Online storage of notes useful for indexing and searching. To be honest - more information goes into this than comes out but it has saved my neck a few times with information I thought irrelevant. I moved from OneNote because there was no suitable OneNote client for the devices I use. Oh and I own a Mac.
TripIt. I'm travelling a bit more these days and it is always nice to have all the necessary information in one place that actually assists planning by proving maps and schedules. Assistive metasites are the future!
Fridge (one of those cold things in your kitchen) - My materialised Kanban board at home that I use for "hobby" activities when I am off work to let my wife know what I am up to and plan accordingly.
Calendar - Outlook synched to Google and available on all devices. I don't use this directly but other people do and I follow the instructions people send to me (if I think it's worth my time).
Email - Most disorganised but oft used medium for retrieving information (usually once and discarded or archived to never be accessed again)

So you see I have got a lot of stuff in a lot of places but luckily important information is exactly where I need it. It may not seem sane or organised but for some reason it is to me - or maybe this is very very normal?

Talking at DDD North and Belfast

Mon Sep 19 23:00:00 UTC 2011

The start of October is going to be a busy time for me - in a good way. The first weekend I'll be doing two talks at DeveloperDeveloperDeveloper in Belfast and the following week I'll be skipping over to Sunderland (first time there) do do another talk for DeveloperDeveloperDeveloper North. I've been lucky enough to land the same presentation for both DDD Belfast and DD North which means I have slightly less to prepare. Not that it will be a drop in quality in anyway - no siree.

For the interested folk the two talks are outlined below. The titles and synopsis are fairly childish/over-sensationalised (delete as you see fit) but thats the way I roll. My opinion enjoys being passed around and so my talks will feature a heavy dose of it - with all necessary disclaimers attached.

For both events I'll be covering,

refORM: Death to ORMs in .NET
Data Access is a difficult area to master. There are plenty of frameworks such as Entity Framework, NHibernate, i/myBatis. These frameworks attempt to make data access simpler but while these products are widely used they can lead to leaky abstractions forcing us to work around constraints of the technology or introduce the nightmare of trying to figure out what is happening deep down in the frameworks internals.
In this session we will look at the various lightweight alternatives and the advantages they offer as well as provide rationale about why the more heavyweight approach isn't always the best approach.

And for DDD Belfast only,

CoffeeScript the Awesome
CoffeeScript - "a little language that compiles into JavaScript" has been making a lot of noise in the web community with its inclusion in Ruby on Rail v3 and its swift rise in popularity on Github.
This session will look at the benefits this language brings over the native JavaScript alternative, the syntactical double rainbow it creates, the problems you'll face and of course how you can actually use it in different technology stacks such as Ruby, node.js, .NET and Java.

I've never really undertaken sessions of this size before (mostly internal or for a customer) so it's going to be a cracking experience. Hopefully people get as much out of these sessions as I will. I'll have whatever resources up after the events as well.

See you there (maybe).

Hackathon #2: Mobile

Wed Sep 14 23:00:00 UTC 2011

So as I've already said in a previous post our first hackathon was well recieved. So why not, we thought, do another one? And indeed why not?

This time we are changing things up. Rather than everyone coming together to bang out a solution we are going to have small teams working on the same project on different mobile platforms. With musical chairs. Yeah you heard right right. Musical Chairs. Kind of.

Heres the deal

A number of stations are set up in a room.
Each station is set up to host a particular development platform (iOS Native, Android Native, PhoneGap, Web, Windows Phone 7 etc.).
Each station will have 2 x Developers assigned to them for pairing and 1 X Technology Expert that will act as an advisory role for that stations technology platform
There will be 1 product owner and 1 idea.
Each station will produce the solution in the best way possible for that platform

Now here's the catch.

Development will be done in sprints lasting 1 hour. At the end of the sprint the 2 devs at each station get up and move to the next station giving a 5 minute handover to the next team. This keeps happneing until the Hackathon (24hrs this time) is over.

Thats the musical chairs bit. But why? Well this allows people to get some exposure to each platform, see the relevant pain points and above all else learn to work as a team rather than being a bunch of selfish pricks (#joke). Sure the first full cycle will probably be painful for most people but like women who live in the same house - after the first cycle everyone should be roughly in sync (#jokeagain).

It's a grand idea but it's easily broken. We need to prevent that happening. Here's how I think we can do that.

People. People are key to this one. More specifically numbers. We need 3 people per station and 1 product owner. If people drop out we have to start cutting stations, if numbers aren't right some teams will be too big/small and that wont work. We got 12 people last time which dropped to about 6 after 24 hours because some people just couldn't hack the pace. this time people need to commit and stay commited or we fail and that would suck!
Planning. Everything needs to be ready for this one. It's got a mobile focus and that should remain the focus. This means all the backend services need to be available and working upfront. This means user stories need to be defined beforehand and even a few broken down into tasks so devs can get cracking immediatley while the rest are fleshed out. This means stations needs to be up and running with everything installed and working. So lots to do.
Server/Services. As I've said these need to be in place and rather than roll our own we are looking at the potentially awesome Parse as it will have everything we need for all platforms thanks to the minimum of a RESTful API. Still under review but it's looking good so far
Idea. We need an idea that is sufficiently small but also feature rich enough to make the implementations different (device integration etc.). We have some but we need to narrow this down to something that would be beneficial.
Infrastructure. All the infrastructure needs to be in place. All dev software installed, source control ready, AgileZen subscriptions bought, test devices ready blah blah blah

I'm sure there's more - I mean it's early days.

So that is where we are right now, lots to think about and lots to do. We are aiming for a November/December slot if we can get the people and a weekend they are all free. Excited++.

Windows 8

Tue Sep 13 23:00:00 UTC 2011

Where WebForms was an attempt to bring the desktop paradigm to web development Windows 8 attempts to do the exact opposite by bringing web to the desktop. More so in fact Windows 8 appears to be effectivley deprecating the desktop idea over time. It's a bold move but trying to cater to every platform is surely a recipe for disaster but you know what I can't help but like it.

I'm not convinced by the Vista v2 arguments either but then I do most of my "poweruser" stuff via a command prompt and thats not going to change - when I hit the desktop I am looking for a nice experience and I feel, so far, I get that with Windows 8.

Being an idiot I managed to grab the ISO without VS11/Blend 5 on it so I am waiting for that one to download before sharing opinions on them (Blend 5 surely must be just like Dreamweaver but I stand to be proven wrong). I am told, however, the first options on the New Project dialog in VS11 are JavaScript projects - I like that. :-)

Tinkering with Agility: Honorable Mentions

Thu Sep 01 23:00:00 UTC 2011

In the process of researching Agility I needed to throw together a quick serverside stack to accept the RESTful calls from Agility. I went with my current framework de jour Tinyweb and I must admit it was a great call. For more basic info on Tinyweb check out my blog post

Agility by default routes the RESTful requests to api/model/{id} and Tinyweb let me create a handler associated with this sort of url by way of areas.

Areas --

Tinyweb has a feature I wasn;t aware of before. It is possible to register a handlers namespace as an "area" (if you have experience with ASP.NET MVC you'll know what that means).

Tinyweb.Areas.Add("Agility.Tinkerings.Web.Handlers.Api", "api");

This means that even though my handler is call TodoHandler the url would map to the one above without having to sacrifice naming style of the handler or force us to create a custom route variable in each handler that had to map to this area. Flippin clever stuff

WebActivator

Another excellent Nuget package is the WebActivator and I have been charmed by it's slickness recently. WebActivator allows you to attribute a class/namespace and specify what code should be run when the application starts. So rather than having to put in a load of lines for configuring different aspects of your stack into Global.asax.cs you can break them into distinct classes and avoid breaking SRP more than you have too. Here is the example I used in the Agility.js source

[assembly: WebActivator.PostApplicationStartMethod(typeof(TinywebActivator), "Activate")]
namespace Agility.Tinkerings.Web.Activators
{
    public class TinywebActivator
    {
        public static void Activate()
        {
            Tinyweb.Areas.Add("Agility.Tinkerings.Web.Handlers.Api", "api");
            Tinyweb.Init();
        }
    }
}

I don't even have a Global.asax defined at all.

Feel free to poke around the source to get a better picture of how I threw the backend together. Thanks go to Tinyweb and @martinrue

Tinkering with Agility

Wed Aug 31 23:00:00 UTC 2011

Having spent a few days tinkering with agility.js I have come to the following conclusions

It's a bit buggy in places (it is currently on v0.1 so this is to be expected)
At first glance code written against agility.js looks like a total mess though it appears to be a weird optical illusion because once you grasp the agility.js approach the code makes total sense
Agility would fit well into an existing codebase as much as into a new one. Certainly more so that some of the other frameworks. And finally,
I like it

You could almost class agility.js as a micro MVC framework (and we all know how I like them). I say that because it takes an incredibly simple approach to MVC as well as offering a very small but (IMHO) complete feature set. Unlike knockout, which also makes use of the data- attributes for model binding (and more), agility.js uses the data-bind attribute for model binding and thats it.

With Agility you have that ability to compose UI's from discrete little self-contained, and/or nested agility objects that behave as standalone MVC implementations. This is kind of different to the standard appraoch of other MVC-like JS frameworks which generally go for the create a single large MVC architecture and manage components within that architecture. I am not saying this appraoch is forbidden or impossible in the other frameworks - it's just not how a noob would go about things. At least I never did when I first started looking at them.

To play with agility I went ahead and created "yet another Todo app" (source) complete with server side persistence and some responsive UI elements (Complete button will vanish when a Todo is completed - wow). I'll not break the entire thing down here but here are some choice cuts (in CoffeeScript cause I am THAT cool)

todo = $$ 
  model: {}, 
  view:
    format: '''
      <li>
        <span data-bind="description"/>
        <button class="complete-task">Complete</button>
        <button class="delete-task">Delete</button>
      </li>
    ''',
  controller: 
    'create': -> 
      @view.$().toggleClass "task-done", @model.get("complete")
    'change:complete': -> 
      @view.$().addClass "task-done"
    'click .complete-task': ->  
      @model.set "complete": true
      @persist $$.adapter.restful, collection:'todos'
      @save()
    'click .delete-task': ->  
      @persist $$.adapter.restful, collection:'todos'
      @erase()

The code above shows the creation of a Todo entity i.e. a single entry in the list. $$ is the Agility factory method and will create an MVC based entity out of the object passed in. I have decalred a view, a model (albeit empty) and a controller with an assortment of tasks for the entity.

This will give you two-way model binding, event wiring up and even custom events totally for free plus a very simple way to react to events. The controller is essentially the event handler so things like click .delete-task will bind the handler to the click event of the element with the delete-task class (the typical full range of selectors are available thanks to jQuery, Zepto integration forthcoming too). Events like create and change:complete are custom events given to us by agility and allow us to listen to key points in the lifecycle of the object. In fact change:complete is a demonstration of something truly awesome (IMHO as usual). The change event will fire when model properties are changed - the change:<property_name> will only fire when the complete property changes. Noice!

My Todo list object then creates instances of the todo object above after pulling the info from the server. It uses agility's inheritance strategy to to create them from the todo prototype I pass in

todos = $$
  model: {},
  view: 
    format: '''
      <div>
        <input type="text" id="newItem" />
        <button class="add-task">Add</button>
        <ol/>
      </div>
    '''
  controller:
    'click .add-task': -> 
      item = @view.$('#newItem').val()
      if item isnt ""            
        newTodo = $$ todo, description: item
        newTodo.persist $$.adapter.restful, collection:'todos'
        newTodo.save()
        @empty()
        @gather todo, 'append', 'ol'

Next I append the todos object to the document (which renders it and wires things up)

$$.document.append todos

Persistance is provided as a plugin of sorts with ability to provide your own adapters. You get a RESTful ajax adapter with Agility. I use this to provide persistence methods on my objects and let agility set that up.

todo.persist $$.adapter.restful, collection:'todos'
todos.persist()

There does seem to be a bug with the factory method and persitance enabled object forcing me to call persist prior to any sort of persitence operation but hopefully we will see that resolved soon.

Finally I fetch from the server and wipe the awesome sauce from my lips and beard.

todos.gather todo, 'append', 'ol'

I've went on more than I wanted to but hey it's worth it. Hopefully this has helped clarify agility to a few people or at least got people intrigued. Feel free to pull down the source and have a go yourself otherwise pop on over to the Agility.js site and read the excellent docs.

Hackathon Retrospective

Sun Aug 28 23:00:00 UTC 2011

So my first ever Hackathon (48hr) is done and dusted and has been for over a week (lazy!!). All in all it was a great experience and plenty of stuff was learnt - we got some technology evaluation (good and bad), ways we could streamline the process and things that burnt time way too fast.

This event was intended as a kind of dry run to see if there is any benefit in Kainos running more events of this nature. Perhaps external ones for charity or educational endeavor. Does that mean I'll be doing it again? Hope so. I guess the best question to ask then is what did I want to get out of it. Well I went in a bit open minded - mainly I wanted to have the experience but I also wanted to try and get some of the techs I have been tinkering with and see how they faired in a semi-real life environment. I think I actually wanted to have produced something really usable too but perhaps I was expecting too much? I mean what we produced wasn't a total failure but it needs a lot of work.

The End Result

We ended up producing a web app for giving continuous feedback to colleagues. The idea was to allow people to give and ask for feedback as well as facilitating a more formal career review workflow. Social appraisals as it were.

We had some basic features such as integrated Windows Authentication and user information look up etc, autocomplete on peoples name, pretty relative time output and some commenting features. So yeah it's pretty sparse but hey it's a start. Whats more interesting is the process and things we observed.

The Story

The team was a mix of Technical Architects, placement students, software and senior software engineers and even a project manager we had a few no shows and a few people who intended to drop in for a while and leave again. We also had a person who thought going out the night before was a good idea - it wasn't (hahaha sucker). But hey at least he showed up. I have no problem with people not turning up as long as they let someone know but not turning up and not telling me…. bad form folks.

Anyway the majority of us gathered in our office at 9ish on Saturday morning and started pitching ideas to attempt. We ended up with a decent pile of good ideas and started aggressively throwing ideas in the bin until, after some debate, we ended up with the one we decided to run with. After that we started thrashing out user stories as a group and ended up with a decent amount. The whole process, including getting food, took about 3 or 4 hours. We then all got stuck into various tasks, I started getting the build server functioning, a few people sat with the product owner and started fleshing out the user stories, a few people looked at the domain model and some people started writing acceptance tests. This was the bit that cost us a lot of time. Rather than each person or group working on vertical slice through the system we went horizontally and started arguing over things that never even made it into the end solution. Everyone fell into their comfort zone cause thats the way Kainos projects typically run - old school waterfall for the most part. So yeah no REAL work actually got done, people were just keen to start and I didn't really pull it under control until it was just too late . It would have been nice to have a Scrum Master type person who was overseeing everything too. I was too busy doing my own horizontal slice :-S.

So things went on like this for a while, a lot of "getting things in place" and some people had to drop off. Eventually we hit a point when we realised that our rate of work was nowhere near what it should be and something needed to change. At about this point everyone but me decided to get some shut eye and I slipped my headphones in, played some Bon Iver and got stuck into fleshing out what we already had. This formed the basis of the work the next day (this was about 4AM on Sunday morning) when the remaining team (5 left out of the initial 12) kicked of the final leg. Due to our earlier stumble there wasn't very much for our product owner/tester to do for most of the day which was a shame. Eventually we ended up with what you seen above (finishing up at about 6PM on Sunday night). The Sunday was less frantic, lots of pairing, lots of trying new technologies and was so much more productive

The Technologies

We used quite an interesting array of technologies

ASP.NET MVC 3
SQL Server Compact 4 (due to issues with SQL Server on the last leg)
PetaPoco
CoffeeScript
Backbone.js
jQuery
jQueryUI
Mig#
Psake
NUnit which I have discovered is a massive impediment to writing tests after the silky smooth experience of NSpec
Should.Fluent
Git

In terms of supporting technologies we used the awesome AgileZen for our Kanban board and HipChat for some shared chatting (we had Lync but it's inability to persist conversations is a deal breaker in my book). Finally, we used TeamCity as our build and continuous integration server which was a much much better experience than CC.NET

Hackathon Review

I guess my story above probably sounds more negative than it really was cause thats the way I write. It wasn't - it was great craic. We held a review post-hackathon with everyone involved and the following point were raised.

Likes

Experience with new technologies e.g. CoffeeScript, PetaPoco
Experience with things such as user stories, kanban etc.
People got to do things that they just don?t get the chance to do. E.g. Andrew was designing screens.
Pair programming
Great craic!

Dislikes

People didn't know what to do when there were bottlenecks
Lunches and dinners were taken at different times. This affected communication.
People thought they weren't productive with no/little sleep.
Tester/Product Owner wasn't used as much as they could've been on the second day.
Facilities room has a motion sensor on the lights
A yoga mat is not good for sleeping on

Improvements

Add an equivalent of a Scrum master to the team.
Agree an idea prior to the hackathon so it doesn't eat into dev time.
Create a couple of use cases going into the hackathon so the dev team don't have to wait on the business people
Set up infrastructure prior to the hackathon
Choose a few technologies before hand so that people can learn a bit about them before. e.g. the software for source control can be decided in advance.
Make sure people aren't going to the shops/getting food at different times
Clone James for tech leader/troubleshooter/developer
Have less team work. We found that the entire team was focused on some tasks that didn't require it because, well, teamwork is great.
Focus on core functionality instead of e.g. logins which can be done later. Or prep login functionality before.
Use hackathon specific machines
If there's another weekend hackathon, people should go get some sleep to rest eyes and minds!

So all in all it was a great learning experience and I don't think anyone said "I'm never doing that again" which was a result and I am certain we will attempt something else in a similar vein and take on board all those improvements to produce something that changes the world or something. So many things to try…. what you all doing next weekend :-P

Life (in IT) Lesson #5372: Learning To Say No, Keep Having Fun

Tue Aug 16 23:00:00 UTC 2011

In case you haven't noticed I didn't blog for nearly a whole month. Thats not like me. Sure, I have my troughs (and peaks) but in terms of my overall social activity I may as well have been mostly dead. My lame excuse? I've been busy. Yeah just like everybody else I've been working more than usual - but I've been taught a valuable lesson. Fun is key.

Over the last month or so I've been gradually building up a collection of in-progress projects - just like Pokemon. I was taking new ones on even when it felt risky and assuring people I could get the work done. My hubris cost me, and others. I spread myself too thin, I left no time for contingency and eventually I ended up, and this is really the best term for it so excuse the language, "fucking over" one of the projects at the worst possible time I can imagine. I've always prided myself on the fact I can always stay on top of an extremely large amount of work and satisfy all the stakeholders. I guess now I know my limit and I even have an indicator well before this threshold is met that I need to start waving a red flag - when it stops being fun.

I enjoy the vast majority of my work, but obviously there are some aspects that less than fun (writing documents, estimating etc.)¹. What started happening was I was doing more non-fun work that fun work (in fact I was doing very little fun stuff at all). This made me less productive, my heart wasn't fully in it and so I wasn't 100% focused. I was less able to predict and plan accordingly and as things started happening on one project (on-site customer meetings etc.) another project had to take the hit and eventually one had to give. The fun factor also meant I was taking on extra work in the vain hope that it might be more fun, which upon reflection, is pretty damn stupid. Oh and lets not forget the fact I was holding off talking to people because of some insane belief that there was a good chance that things would look much better in "another hour or so". Seldom was this the case of course so it mean't that I wasn't raising flags early enough and people felt I was being aloof.

So what will I be doing in future? Well I now have a rule and I will see how it goes. Simply put,

I will only take on more work if I am currently having fun and the new work will not push me into the not fun work spectrum. If I am not having fun I can only take on more work when I have cleared some of my work and am back having fun. The closer I get to the not fun end of the spectrum the sooner and more frequent communication with stakeholders needs to be.

Simple enough but it seems to hold up so far.

Give me too much fun work and it stops being fun because there is too much pressure.

Give me a single not fun job that isn't fun and I will only accept it provided I can work on my own projects to re-address the fun ratio (blogging about experiences, related technologies etc.)

This probably sounds quite selfish and, well, of course it is. Nobody should expect me to just do whatever they say² without at least making the deal appealing to me or at least allowing me a certain amount of free time to draw my own benefits from the situation. For example I accept a certain amount of not fun work because I am permitted to do a lot of my own fun work to address the balance.

I think I've left this open enough for personal interpretation and flexibility but strict enough that there is a foundation here to aid in making decisions. If I had applied this rule previously I wouldn't have taken the extra work (even though it seemed appealing at the time - it was fun and I got paid overtime for it) and both myself and the project would be in better shape. But hey how can you ever learn if you never make mistakes?

Is the word fun starting to sound really weird to anyone else?

¹ I realise some people LOVE writing documents and estimating - I don't. They are just a necessary part of the job that I live with.

² There is only one person that CAN - my wife and thats like the law. The company I work for is great at trying to keep their workforce happy and I am in no way implying that they forced work on me. They didn't.

An Opinionated Project Structure

Mon Aug 15 23:00:00 UTC 2011

Choice and flexibility are good, in fact they a super smashing awesome. Sort of. I mean if you go over the top and offer people "all the choices" they are going to get all stressed and probably procrastinate until someone else makes the decision for them. Also if you give them so much flexibility they will probably do things differently each time - again that is not a great thing either - for many things in life and IT you shouldn't have to make a conscious decision (it's probably waste of time in this case).

I've recently been reviewing my companies default project structure and it's full of choice and flexibility and while this can cater for the vast array of different project types in our company it also imposes a certain level of cognitive-tax prior to kicking off any development to get the solution in a position that can be farmed out to the team. It imposes this tax across ALL projects even the 99% of the time typical project, with their typical project structure. It's not a costly tax but it's unnecessary. Why not just push it onto the non-standard projects who will likely need to spend time setting things up correctly anyway? Suck it, weird projects! Thats what I say.

To this end I've been trying to build a fairly opinionated project structure (along with some handy tools during the development stage). Right now it's in the early stages and open to abuse/commentary (in fact I'd welcome it, please) but I thought it makes sense to share it¹ - for the benefit of myself and others (or selfishly unselfish as I like to call it).

Folder Structure

I didn't want to break from tradition here. There is exists a fairly standard project structure that is common across many languages and environments and rather than create a clever, verbose structure I reckon it's probably best to stick with the tried and tested approach. That way new developers, regardless of background, shouldn't have to wrestle with heavily nested folders or obscure names for folders (SolutionSource vs src) when navigating our solution.

|_ <MySolution>
    |_ <lib>
   |_ repositories.config
    |_ <src>
    |_ <test>
 |_ .gitignore   
  |_ default.ps1  
  |_ MySolution.sln
 |_ nuget.config

Most of this will be fairly self-explanatory² but whats the harm in a little clarification?

lib this is the place either myself or Nuget (see nuget.config and default.ps1) will dump any DLLs, tools or other dependencies required by the solution. As I will talk about later the only file I ever check in here is the repositories.config and have nuget resolve the rest (still some decisions need made around manually referenced files etc.).
src hold all the projects that are actually going to be released as part of the final solution. This folder doesn't contain any of the test projects you create for the solution - no point in muddyinbg the waters.
test holds all the solutions projects that we create for running tests (unit, integration etc) to validate the solution.

Special Files

My default project comes with a number of files that help make developing and building the solution form scratch as easy as it should be. These files are open to be tweaked to suit the needs of the project as and when necessary.

repositories.config

This is a file that is used by Nuget to resolve the location of all the package configuration files associated with the projects in a solution (packages.config). When I start this file is empty but as dependencies are added via Nuget this file is updated to the location of packages.config files within each project (which in turn is used to resolved the necessary Nuget packages that need to be downloaded). I include this file along with a download task in my psake build file (see later) so that we we don't have to check-in all those, potentially large, dlls and tools into our VCS - it's slow and annoying. As a bonus when using some VCS's (e.g. git) this file will ensure the lib folder is committed to source control even thought it is technically empty - keeping our project structure intact.

.gitignore

Used to ignore certain generated and user specific files when using the git version control system. Keeps things nice and clean and reduces annoying unnecessary conflicts. As with most solutions out there I ignore the following patterns by default,

bin
obj
*.suo
*.csproj.user
*.cache
lib/*/

nuget.config

I'll jump onto the meat of the files, default.ps1, shortly but first as you can probably guess by now I assume the use of Nuget. It's there, it's great and thats all I want to say on that matter. nuget.config is a simple file that tells Nuget (even within Visual Studio) to put all it's downloaded files into the lib directory rather than the default packages folder. Everyone understands what lib is, less people - packages.

<settings>
 <repositoryPath>lib</repositoryPath>
</settings>

default.ps1

I've decided to run with psake to coordinate and perform my builds. MSBuild, while useful, is ridden with sickeness I call XML (BTW everyone else calls it XML too AFAIK). It's a beast to get right. Psake on the other hand is pure script. It's more succinct and you can even debug it - how awesome is that? So default.ps1 is my psake based build file. What does it look like currently? Glad you asked.....

# --
#                     S C R I P T   P R O P E R T I E S 
# --
Properties {
 $solution = (Get-ChildItem *.sln).Name
  $solutionname = $solution.Substring(0, $solution.LastIndexOf('.'))
}

I assume that the majority of projects are based on a single solution held at the root of our folder structure (as described above). So I dynamically grab the name and relative path of the solution file for use later.

# --
#                      H E L P E R   F U N C T I O N S
# --
function Write-LineBreak(){
  Write-Host "-"
}

Set of helper functions currently only provides a simple function to print out a flowerboxing line to make output a bit cleaner.

# Set the default framework (3.5 by default)
$framework = '4.0'

Assume that we are all living in the present (as much as possible) so switch to .NET 4.

# Declare default task
Task Default -depends Build,Test

By default psake should do a Build and Test.

# Resolves all the requires nuget dependecies so they don't need checked in
Task Resolve {
  Push-Location '.\lib\'
  Get-Content '.\repositories.config' | foreach {    
 if($_ -match 'path="(.*)"'){
  
      Write-LineBreak
     Write-Host "Resolving Dependencies for " -nonewline
     Write-Host $matches[1].Split('\')[-2] -ForegroundColor Green
      Write-LineBreak
 
      nuget install $matches[1]
   
      Write-LineBreak
 }
  }
  Pop-Location
}

Resolve is the function mentioned earlier that finds the location of the repositories.config files and asks Nuget to resolve all the external dependencies of the solution. This should be done when you have pulled from a repository that included new dependencies.

# Build the first solution file you find
Task Build {
  
  Write-LineBreak
  Write-Host "Building Solution " -nonewline
  Write-Host $solutionname -ForegroundColor Green
  Write-LineBreak
      
  Exec { msbuild   /v:quiet /t:Rebuild  }
  
  Write-LineBreak
}

Performs a basic, quiet build of the default solution.

# Run the project tests
Task Test {
}

Executes all the tests in the solution. Currently empty I intend to flesh this out with a default unit testing approach.

Baby Steps

So this is just the beginning and I hope this evolves based on observations made in our real world projects rather than a bunch of nice to haves. It will also be nice to automate the creation of projects and solutions but thats certainly for another time. Recommendations, comments and all that stuff welcome. I'll try and get this basic structure up on Github too.

¹ I mean, it has been a while since I blogged so it's about time I get back to it. That said this isn't a "filler post". I'm not doing this under some random contract. Thought I wish I was - imagine that - how fun.

² Cracker idea, right? Who'd have thought it??! I'm a total genius and you can bow at my feet at will please. kthnxbai.

³ All subfolders in lib - as mentioned in the previous section.

Micro Web Frameworks in .NET 101: Tinyweb

Sun Jul 17 23:00:00 UTC 2011

Tinyweb takes a slightly different view than the other web frameworks I have talked about. In fact it takes a fairly opinionated approach to the way your structure your applications code and forces you to think in terms of resource endpoints rather then big monolithic modules or controllers. It's certainly true that projects based on the other frameworks such as Nancy and Jessica can be architected in such a way but Tinyweb ensures that you don't start cutting corners and making allowances for lazy code by simply not providing the ability to do it!

Getting Started - Hello World

Lets get Tinyweb first from good old reliable Nuget,

Install-Package Tinyweb

With Tinyweb installed we can go ahead and create a RootHandler this name is the one exception to the Tinyweb handler naming convention which I'll touch on in a minute.

public class RootHandler
{
    public IResult Get()
    {
        return Result.String("Hello World");
    }
}

In our handler I also defined the a Get method which returns an IResult this will act as our endpoint. One last thing before we magic up a web page is to initialise Tinyweb via our Global.asax.cs so Tinyweb can do its bootstrapping discovery voodoo stuffs.

public class Global : HttpApplication
{
    protected void Application_Start(object sender, EventArgs e)
    {
        Tinyweb.Init();
    }
}

Et voila! Run the app and once again we have a hello world.

Tinyweb Features

Even in that small Hello World example there are plenty of Tinyweb features but you'll also notice that Tinyweb really doesn't get in the way at all.

Handlers

Each Tinyweb Handler represents a single resource endpoint. With the exception of RootHandler the URL endpoint for each handler is inferred from the name of the handler. For example a handler named HelloHandler will react to /hello, HelloWorldHandler will react to /hello/world. You can see from these examples that casing of the handler name is important in terms of the url endpoint generation.

The next thing about handlers is that they respond to 4 methods each one corresponding to the 4 main Http Verbs - Get(), Post(), Put() and Delete - the only requirement of these is that they return an IResult

Model Binding and Arguments

Model binding is also provided by handlers actions - just pass in an object and Tinyweb will do it's best to bind request values to this object (accepts primitives, collections and plain C# objects).

Alternatively if you want to do some work under the hood you can make sure the RequestContext is passed in by simply passing it instead. This gives you access to the bowels of the request to do with what you please.

Advanced Routes

It's also possible to override the default routing convention by declaring a handler level variable called route of type Route. This example shows how we can override handlers default route /hello/world and go with the more understandable "/helloworld".

public class HelloWorldHandler
{
    Route route = new Route("helloworld");

    public IResult Get()
    {
        return Result.String("Hello World");
    }
}

Route Parameters

In the spirit of RESTFulness you may also want to accept parameters as part of your URL and we can do that as well with this route class as well as optionally suppling default values for parameters,

public class HelloWorldHandler
{
    Route route = new Route("hello/{name}");
    // OR WITH A DEFAULT VALUE FOR PARAMS
    Route route = new Route("hello/{name}", new { name = "World" });

    public IResult Get(string name)
    {
        return Result.String("Hello " + name);
    }
}

A powerful little approach.

Results

All Tinyweb results (e.g. what the response returns) implement the IResult interface so it's simple enough to implement your own custom result type though it is probably unnecessary for the most part as Tinyweb offers a range of result types straight away via the Result classes static methods.

String
File
Json
XML
JsonOrXml (returns either Json or XML depending on the request headers)
Html
Redirect (to a specific handler or URL)

Tinyweb also offers a View class that can render views written with Spark and Razor so rendering a view can be as simple as

public class HelloWorldHandler
{
    Route route = new Route("hello/{name}", new { name = "World" });

    public IResult Get(string name)
    {
        return View.Razor<string>(name, "hello.cshtml");
    }
}

Filters

Filters allow us to intercept requests both before and after they are processed on both a per handler and global level.

Each handler can optionally contain an After and/or a Before method that will, not surprisingly, be called after and before each handler request (I'll let you guess which one does which) e.g.

public class RootHandler
{
    public void Before()
    {
        Logger.Log("Before Executed");
    }

    public IResult Get()
    {
        return Result.String("Hello World");
    }

    public void After()
    {
        Logger.Log("After Executed");
    }
}

Alternatively if you want the same before or after filter applied across all handlers you can create a Filter class by creating a class appended with the word Filter. We can recreate the same handler above with a global filter for logging,

public class LoggingFilter
{
    public void Before()
    {
        Logger.Log("Before Executed");
    }

    public void After()
    {
        Logger.Log("After Executed");
    }
}

public class RootHandler
{
    public IResult Get()
    {
        return Result.String("Hello World");
    }
}

Before and After methods can also return IResult objects if you want to work with the actual response.

Error Handling and Debugging

Not all of us write flawless code and so sometimes things go south and exceptions start throwing their weight around. Tinyweb has a global hook that is useful for capturing such errors as they bubble up to the surface. The TinyWeb class has a static property OnError that accepts an Action that can be used to handle the exception e.g.

protected void Application_Start(object sender, EventArgs e)
{
    Tinyweb.Init();
    Tinyweb.OnError = (exception, context, data) =>
    {
        Logger.Log(exception);
    };
}

Another useful tool when debugging your app is the Tinyweb.WhatHaveIGot(). It's a convenience method that can be used to print out all the matched routes and filters.

Conclusion

On the surface the focus of Tinyweb may seem only moderately different from other frameworks but after playing with it for a while it become apparent that the opinionated approach really makes you think about your projects structure and API. I'm certainly of the opinion that this is a damn good thing¹. Tinyweb gives me just enough framework to do pretty much everything I need - which I would expect from a real micro-framework. On top of that it's flexible and intuitive and another handy utility in my tool belt.

¹ I've already said that it is possible to make the likes of Nancy and Jessica behave in this, either by ensuring that all module adhere to this pattern, or creating an abstract base class that enforces this sort of structure so I am in no way bashing the alternatives. I like choice.

Baler v0.4.1 Released

Sat Jul 09 23:00:00 UTC 2011

Tiny release push out to Baler that provides VS2008 specific project and solution files. No new stuff pushed to Nuget as it is source only this time.

Source available in the Github Repo.

Get Excited and Make Things

Tue Jul 05 23:00:00 UTC 2011

Not generally a fan of the Keep Calm and Carry On parodies but this one pretty much sums up how I feel about my job/career type thingies and well worth a mention. Props to Matt Jones for this one.

Debugging NSpec Tests: The DebuggerShim

Mon Jul 04 23:00:00 UTC 2011

Busy busy busy. I've been working away on a lot of projects and not had a lot of time to stick anything up on this blog in a while. I've got a growing "To Blog" list but no time to do it, woe is me!!!

Enough about my perfectly normal life, well almost. Some of my current work has me going full tilt with TDD on a little C# project and I decided that NSpec would help me with unit testing. So I went about writing failing tests, writing code, passing tests on and on and on. Things went along fairly well until I hit a little bit of a wall around a little edge case. My tests kept failing and I couldn't understand why. The stack trace made little to no sense and my code looked sound. The problem I had was that NSpec, out of the box at least, doesn't have any nice integration with Visual Studio or TestDriven.NET. Initially I was using a throw away console application and debugging through it but as you can imagine that was time consuming.

It wasn't long though until I was pointed in the right direction.

If you want to have debugger support for you NSpec specifications, use this: http://t.co/6SwZcVL /cc @kouphax @mattfloless than a minute ago via web Favorite Retweet ReplyAmir Rajan
amirrajan

The gist, in full below, provides a simple debugger shim over nspec that allows you to hook into the Visual Studio debugger (through something like TestDriven.NET for example). I simply right clicked on the shim and select "Run With Debugger" and boom my stupid mistake was displayed to me rather quickly.

using System;
using NUnit.Framework;
using NSpec.Domain;
using System.Reflection;
using NSpec;

namespace DynamicBlog.Tests
{
    [TestFixture]
    public class DebuggerShim
    {
        [Test]
        public void debug()
        {
            //the specification class you want to test
            //this can be a regular expression
            var testClassYouWantToDebug = "describe_Blog";

            //initialize NSpec's specfinder
            var finder = new SpecFinder(
                Assembly.GetExecutingAssembly().Location, 
                new Reflector(), 
                testClassYouWantToDebug);
            
            //initialize NSpec's builder
            var builder = new ContextBuilder(
                finder, 
                new DefaultConventions());

            //this line runs the tests you specified in the filter
            new ContextRunner(builder, new ConsoleFormatter()).Run();
        }
    }
}

Handy until a more concrete solution appears. One more thing - really enjoying nspec, it seems to be fitting well with the way I work.

Micro Web Frameworks in .NET 101: Anna

Tue Jun 21 23:00:00 UTC 2011

It's been a while since I visited this subject (at least in my blog) and I've still to record a lot of screencasts around Nancy. Anyway I thought I'd give a quick mention to Anna that has just come up on my radar. Now the project is still in the very early stages but it's possibly one of the best uses of Rx and worth a quick look. Now I am a bit rusty in the old Rx stuff so sorry if I say something stupid.

Getting Started - Hello World

Anna aims to be an asynchronous event-driven HTTP server in a similar vein to node.js. At it's core is the HttpServer object that allows us to wire up routes.

There is no configuration required - just install Anna from the Nuget Package Console via Install-Package Anna and you are good to go. Just create a console application with the infamous Program.cs and a few lines of code gives you the Hello World solution,

using (var server = new HttpServer("http://localhost:1234/"))
{
    server.GET("/").Subscribe(context => 
        context.Respond("Hello World"));
    Console.ReadLine();
}

I added the Console.ReadLine() to stop the console app exiting and cleaning things up. Running the app and going to http://localhost:1234 gives you the result. The HttpServer object has a number of methods for the various HTTP verbs and these methods all return instances of IObservable which means you can subscribe to them and apply all those filters, transforms etc. the Rx offers.

Other Anna Features

As I said Anna is still in the early stages so once you grab a route there isn't a MASSIVE amount you can do with but respond. Responses however are chainable so you can create an almost middleware like stack for requests e.g.

var route = server.GET("/");
route.Subscribe(context => context.Respond("Hello "));
route.Subscribe(context => context.Respond("World"));

This gives you the same output as the original Hello World sample.

But from what I can see thats about as far as you can get. In node.js there is a framework called Connect now I'm not sure if Anna is intending to go in this sort of direction but I think it could be moulded into something like this with a bit of TLC. The problem is at the minute I don't see any sort of object passed along the subscription chain e.g. a dynamic bag or something. I don't even know how to easily manage response headers.

Routes and Route Fragments

Anna supports creating different routes including routes with dynamic sections /name/{name} that are used to populated a UriFragments property,

server.GET("/hello/{name}").Subscribe(context => context.Respond(
    string.Format("Hello {0}", context.Request.UriArguments.name)));

Conditional Routes

Thanks to the power of Rx and Linq it is possible to create conditional routes an create chains that have pre-response checks. For example we can have a route that only triggers a response if the QueryString has a secret parameter.

var route = server.GET("/");

route.Subscribe(context => context.Respond("<h1>Hello World</h1>"));
route.Where(ctx => ctx.Request.QueryString.secret == "letmein")
     .Subscribe(ctx => ctx.Respond("<h2>Secret Message</h2>"));

So if you hit http://localhost:1234/ you get Hello World. Hit that again with the param ?/secret=letmein and you get the super secret message.

Conclusion

Thats about it for now but I think I'll be keeping an eye on this project and see were it goes. Even though it's still early days I like the concept behind the library and the fresh approach it's taking. I'll push some of my sample code up to Github and update this post in the next few days. In the meantime Jose the creator of Anna has some nice examples of Rx based HttpServer stuff on his blog featuring a long polling example.

MicroORMs for .NET: Inserts, Updates & Delete

Mon Jun 20 23:00:00 UTC 2011

This post is the is part of series of posts covering the various microORMs in the .NET world. I intend to look at how each microORM meets certain data access and manipulation needs. The current series includes,

Obligatory Introduction Post

Syntax - SELECTs

Stored Procedures

Inserts, Updates & Deletes

Moving swiftly on, if all our solutions only used different way to select data then how would we ever get data into our solution? So lets look at how these microORMs handle Inserts, Updates and Deletes¹. These examples will follow on from previous ones so I am only going publish the key lines. If you want the complete code you can grab them from the GitHub project.

Dapper

Insert

Nothing unexpected here. More SQL. Sam Saffron previously pointed me in the direction of Dapper.Contrib that has extra extension methods for doing things like INSERT, UPDATE and DELETE but at the time of trying I couldn't get it to work and have an open issue on GitHub. If I get an update on this I'll post the slightly cleaner syntax.

conn.Open();
int count = conn.Execute(
    "INSERT Authors(Username, FullName, CreatedDate) " +
    "VALUES (@Username, @FullName, GETDATE())",
    new { Username = "james@dapper.net", FullName = "James Hughes" });

Update

Update is very similar to insert. The nice thing about this approach vs. using POCOs is that you only need to send the delta of the object up where as with the other approaches you may have to fetch the object before updating.

int count = conn.Execute(
    "UPDATE Authors SET FullName = @FullName WHERE Id = @Id",
    new { FullName = "James Hughes", Id = 1 });

Delete

Delete is pretty much what you'd expect. I'll update these examples when I get my problems with Dapper.Contrib sorted but when you are so close to the SQL metal there isn't a massive amount to discuss.

int count = conn.Execute(
    "DELETE FROM Authors WHERE Id = @Id",
    new { Id = 1 });

Massive

Insert

Massive keeps things simple as usual. Insert can take POCOs as well as dynamic object (and a number of other types) and it uses the info found in the Authors DynamicModel class to generate the SQL for you.

var tbl = new Authors();
var x = tbl.Insert(new 
{
    Username = "james@dapper.net",
    FullName = "James Hughes",
    CreatedDate = DateTime.Now
});

As a bonus feature Massive can wrap multiple inserts in a single transaction and perform multiple inserts at once.

var tbl = new Authors();
var x = tbl.InsertMany( new [] {
    { Username = "james@dapper.net", FullName = "James Hughes", CreatedDate = DateTime.Now }, 
    { Username = "james@dapper.net", FullName = "James Hughes", CreatedDate = DateTime.Now }, 
    { Username = "james@dapper.net", FullName = "James Hughes", CreatedDate = DateTime.Now } 
});

Update

Like Dapper, Massive allows us to send only a delta between the old and new object.

var tbl = new Authors();
            var x = tbl.Update(new
                                 {
                                     FullName = "James Hughes"
                                 }, 1);

Massive also gets extra points for the UpdateMany that, not suprisingly, behaves like the InsertMany method mentioned above (except for updates of course).

Delete

Ultra simple approach to Deleting specific objects just pass in an id and you're done,

var tbl = new Authors();
var x = tbl.Delete(2);

PetaPoco

Insert

PetaPoco offers 2 main ways to insert your data². You can pass in a plain POCO (or in our case on that uses attributes to normalise the difference between the schema and the C# class),

PetaPoco.Database db = new PetaPoco.Database("DefaultConnectionString");
var x = db.Insert(new Author
{
    Username = "james@dapper.net",
    FullName = "James Hughes",
    CreatedDate = DateTime.Now
});

Or you can pass in an anonymous object specifying the table and primary key field,

var x = db.Insert("Authors", "Id", new
{
    Username = "james@dapper.net",
    FullName = "James Hughes",
    CreatedDate = DateTime.Now
});

Update

There are plenty of ways to update data using PetaPoco. The first one is to pass in your model object and the row will get updated. I am creating a new object in this example but you could just fetch it from the database as you'd expect,

PetaPoco.Database db = new PetaPoco.Database("DefaultConnectionString");
var x = db.Update(new Author
{
    Id = 9,
    Username = "james@dapper.net",
    FullName = "James Hughes",
    CreatedDate = DateTime.Now
});

Another approach is to just pass up a delta of the changes,

var x = db.Update("Authors", "Id", new
{
    Id = 10,
    FullName = "James Hughes"
});

Alternatively you can simply pass up the delta and specify the primary key value externally,

db.Update("Authors", "Id", new { FullName = "James Hughes" }, 12);

Variety is the spice of life in PetaPoco!

Delete

Thanks to the custom attribute on our Author class deleting a record is a matter of passing the Id to the Delete method,

db.Delete<Author>(8);

ServiceStack ORMLite

Insert

ORMLite is pretty much inline with the other microORMs a simple Insert method that accepts a POCO or an anonymous object and generates SQL based on the info given. This can be done in one of two ways,

using (IDbConnection db = Program.ConnectionString.OpenDbConnection())
using (IDbCommand cmd = db.CreateCommand())
{
    cmd.Insert<Author>(new Author
    {
        Username = "james@dapper.net",
        FullName = "James Hughes",
        CreatedDate = DateTime.Now
    });
}

Or the factory approach discussed in the first article,

IDbConnectionFactory dbFactory = new OrmLiteConnectionFactory(
    Program.ConnectionString,
    SqlServerOrmLiteDialectProvider.Instance);

dbFactory.Exec(dbCmd => dbCmd.Insert<Author>(new Author
{
    Username = "james@dapper.net",
    FullName = "James Hughes",
    CreatedDate = DateTime.Now
}));

Update

ORMLite sticks with one approach for updating a row - passing in a POCO that it can map to a table/row,

cmd.Update(new Author
{
    Id = 7,
    Username = "james@dapper.net",
    FullName = "James Hughes",
    CreatedDate = DateTime.Now
});

If you really don't want to fetch your row again and only want to send up a delta you'll have to roll your own approach (simple enough though).

Delete

ORMLite supports a number of neat ways to delete rows,

DeleteById(id)
Delete(where_clause, id)
Delete(object)

cmd.DeleteById<Author>(5);
cmd.Delete<Author>("Id = @0", 6);
cmd.Delete(new Author { Id = 7 });

Simple.Data

Insert

Guess what Simple.Data's approach is once again very fluent, clean and easy to understand. OK I don't get Intellisense as the methods are dynamic but I didn't need it anyway.

Author x =  Simple.Data.Database.Open().Author.Insert(new Author
{
    Username = "james@dapper.net",
    FullName = "James Hughes",
    CreatedDate = DateTime.Now
});

Update

Simple.Data has 2 ways of updating. Update takes a POCO and maps the id to the specific row,

Simple.Data.Database.Open().Authors.Update(new Author
{
    Id = 5,
    Username = "james@dapper.net",
    FullName = "James Hughes",
    CreatedDate = DateTime.Now
});

Alternatively Simple.Data supports the delta approach too using UpdateById

Simple.Data.Database.Open().Authors.UpdateById(
    Id: 6, 
    FullName: "James Hughes");

Delete

Finally Simple.Data has 2 ways to delete a record,

DeleteById(id)
Delete(named_parameters)

Simple.Data.Database.Open().Authors.DeleteById(3);
Simple.Data.Database.Open().Authors.Delete(Id: 4);

There We Have It

OK OK I guess this post feels a bit rushed. I started out throwing lots of details into selects and stuff but this one just isn't up to par. I slowly discovered that I'd have to spend almost all my spare time covering all the aspects I wanted. So I cut it back to a kind of "Starter for 10" approach. Yeah thats right I'm leaving stuff up to you to ask and research :-P. Anyway in these basic examples no one ORM really stands out too much. Simple.Data is, as always, lovely and clean and PetaPoco offers a very flexible experience. I like the ability to send deltas instead of full objects back (in case you hadn't guessed) and I had a few annoyances around updates using Massive and Dapper but I think it was my own stupidity (I'll update when I investigate). If I had to choose between them I am still tending towards Simple.Data and PetaPoco (probably PetaPoco at this stage) but I do find all of them a much more pleasant experience vs. MyIbatis or NHibernate.

That should be enough to start anyone off on the right foot.

¹ Hehehehe IUDs

² You can pass a SQL string or a SQL builder as well but it's fairly straightforward so I'll not cover it here. See the original SELECT post for a SQL and SQL Builder example.

Baler v0.4 Released

Sun Jun 19 23:00:00 UTC 2011

Just a quick note to say that Baler v0.4 has been released. Nothing new here I have simply compiled it (and the official extensions) against .NET3.5 just to make it more useable for people still stuck on old projects :-P

Everything is on Nuget already so go have fun.

Thinking about Hackathons

Sat Jun 18 23:00:00 UTC 2011

This post is a bit of a brain dump so expect some incoherence and repetition. I'm buzzed on coffee and feel like a kid in a toy shop so I just want to get it out there and if anyone has any experience of hackathons feel free to pitch in. In fact please do :)

Also sorry for the blogging overload.

Background

So I recently started walking to and from work. The walk is about 3.5 miles which takes about 50 minutes. This gives me a lot of time to think, even over think but even I bore myself sometimes so I decided to give podcasts a try. I grabbed a load of Hanselminutes and Herding Code episodes to maybe learn something new. One of the Herding Code podcasts really caught my attention - 109: Harmony Hackathon. The podcast tells the story of a group of people that got together and spent 48 hours designing and developing a solution for a charity organisation. I've heard of hackathons before but hearing these guys talk about their experiences while using a technology stack that is my bread and butter (.NET) just got me all tingly inside.

I wanted to do this. I know it sounds pathetic but the experience of hashing something out, cutting code and working totally agile without the bureaucracy of management slowing things up¹ sounds really fun to me. I tend to get bored on long running projects. I really want to try new things in real projects but convincing managers or customers is a near impossible task as the "play it safe" attitude makes a lot more sense to them. Come on guys who cares about money when there is fun to be had?

So anyway this got me thinking. How would I go about this, why would I go about this and how would I get other people to give 24 or 48 straight hours of their life for something like this? I mean I live in Belfast, Northern Ireland not exactly a MAJOR hub of geek activity.

Hackathon Goals

So why would we do this? Having clear goals upfront is really going help sell this to people. They will also act as guidance and help drive the entire event to the finish line. There are plenty of reasons, perhaps even contradictory ones,

A finished product or prototype
Education of technologies and techniques
The experience
Technology evaluation
Networking
Something to put on your CV

I realise that much of this is done on a per person basis but unless each persons goals are aligned in some way there is a risk the whole thing would fall apart. Personally I think people wanting to leverage it for any sort of networking, CV building or taking unfounded punts on technology need not apply. I am not adverse to a bit of risk taking in the technology stakes but they need to be calculated risks rather than a "ohhh shiny lets use this" attitude. If it's putting all the other goals at risk it's not worth it.

A finished product or prototype
Education of technologies and techniques
The experience
~~Technology evaluation~~
~~Networking~~
~~Something to put on your CV~~

I'd be happy working with people who agree with me on at least 2 of the points above. But of course this isn't a hard and fast rule and I remain open for the greater good.

The Idea

So what would we do? I don't think it would work very well if we just played around with some technologies and produced a few prototypes. No, addressing a real work problem or need that is of sufficient size and complexity that allows us to drive a spike right through the entire solution gives us something to work towards. Actually producing something that works or at least proves that the idea is viable seems almost mandatory. The thought of walking away after the 24 or 48 hours with something that you can show people is a noble goal. Whether that be for charity, something internal for your company or something for the open source community.

I have plenty of ideas I would go for but this isn't a dictatorship (not at this point anyway) this is only going to work if people feel really involved, after all when the "mid way slump" starts kicking in you're only going to make it through if you WANT it to work and are invested in experience. I guess what I am saying is you'd get people together - potential developers, product owners, investors up front and brainstorm, plan and argue. This brings me on to another point….

The 7 P's

Prior Preparation and Planning Prevent Piss Poor Performance. The more I think about this hackathon thing the more I come to the conclusion that it is so much bigger than the 24 or 48 hours of design and development. Like all good adventures you have to go in prepared. What would you need upfront?

You need your goals clearly defined
You need a few user stories so there is something for people to do while things are getting established
You need your stack and approach mapped out (including source control, CI and Build Server etc.)
The participants need to at least have a fair idea what they are doing
You need to have some sort of rough plan or schedule
You need food, drink, beer, alternative entertainment
You need a venue, a comfortable venue with internet access
Oh and I guess you need to invest in whiteboards. Lots and lots of whiteboards, and markers, and cue cards and possibly deodorant.

People involved

Agile makes sense here, of course it does. So it's not just a matter of locking a few devs away with coffee and vitamin pills. No you need product owners at hand, you need someone testing and you need some sort of BDFL (Benevolent Dictator for Life). I guess a BDFL isn't mandatory but a technical lead who has the final say is certainly going to make the decision making process a lot quicker - be that good or bad.

Motivational people are another thing that would come in handy. Having people see something and be like "yeah that's nice, good job" is going to keep those flagging energy levels up.

Developer headcount is another balancing act. You need enough people to make the goals achievable but you also can't have too many people or you risk stamping on each others feet. I guess it depends on the size/complexity of the idea you want to implement but a finger in the air guess I'd say between 4-8 developers with a product owner and a tester (could be the same person) would be a nice number.

Development Approach

I've already said agile seems like a perfect fit and I think the whole ecosystem that agile brings in fits well. TDD/BDD almost seems mandatory here - at least some level of it. Checking in code with no tests around it is going to cause a headache when you hit a bug that takes and hour to track down. A strict regime of constant branching, merging and checkins (every 20 minutes or less) along with a fast continuous build system will help discover problems ASAP.

Atmosphere and Communication

An entire project condensed into 24/48 hours is going to magnify the pain points and tensions will rise. Techniques such as Pomodoro would come in handy to make sure people aren't burning themselves out too fast. A quiet room and alternative non-computer based entertainment will give people a quick escape on their pomodoro break and a chance to get their head showered. Communication NEEDS to flow so regular standups (every 30 minutes to an hour) are essential to spot problems early and find a way to resolve them quickly. Again having some sort BDFL could really help in this situation.

Where now?

You know what - someway, somehow I am going to do this. Consider it part of my career bucket list. Something I want to do before I join the great geek scrapheap in the sky. Could I get the people? Yeah I reckon I could. Could I get the sponsorship/support? Most likely. Could I find a suitable idea to run with? Hells yeah. So what is stopping me? Hmmmmmm nothing. BRB got a hackathon to organise……..²

¹ Not all managers are created equal but they have a job to do and process, meetings, documents etc. all get in the way.

² Yeah really. No joke. I can but try.

MicroORMs for .NET: Stored Procedures

Fri Jun 17 23:00:00 UTC 2011

This post is the is part of series of posts covering the various microORMs in the .NET world. I intend to look at how each microORM meets certain data access and manipulation needs. The current series includes,

Obligatory Introduction Post

Syntax - SELECTs

Stored Procedures

Inserts, Updates & Deletes

As handy as running straight, fully-tailored SELECTs straight from your app code a lot of the time, especially in the "enterprise" world your DBA is going to insist that all operations go through tightly controlled stored procedures. It's all for the greater good I'm sure but flipsake it's bloody annoying.

You are standing at the edge of a forest.  Ahead of you an old wooden sign sits at a slight angle.
>> read sign
You squint your eyes as you approach the sign and read the fading writing.  The sign reads
"Welcome to the forest of microORMs.  You have nice eyes.  Have a GREAT day."
>> enter forest
You walk, nay skip, into the forest.  A few minutes in you are confronted with a horribly gnarled beast.  The grumpy looking thing seems very out of place in this land of near tranquilty.  As you approach, it speaks
"GROAN GRUNT Control GROWL Best Practise WHEEZE Enterprise Ready SNARL.  Little being... you may not pass any way you please.  That is not how we do things GRUMBLE WHEEZE. No to pass this way meatbag you must, yes must, take this route I have planned for you.  If you need to change your route I must approve it and construct it COUGH GRUMBLE.  There is limited scenery and not too many points to have fun"
>> kill beast
I'm afraid I can't do that
>> fffffffffffffffffffff..

Fear not help is at hand. So can we call sprocs using our microORMs, and if yes how? Lets go see shall we?¹

Dapper

Dapper makes use of an extra parameter in its Query<T> method to specify a specific command type. In this way it's simply a matter of passing in the name of the sproc, the parameters and a CommandType.StoredProcedure as a named parameter.

Author author = conn.Query<Author>(
    "GetAuthorById",
    new { Id = 1 },
    commandType: CommandType.StoredProcedure
).First();

And boom thats it. Same applies for multiple results.

Massive

Massive just works.

dynamic dyn = tbl.Query("GetAuthorById @0", 1).First();

All I did was swap out the SELECT statement for the name of the sproc and Bobs Your Uncle. Nice.

PetaPoco

PetaPoco has some support for sprocs. For my needs in this article it works fine but apparently it doesn't support output parameters ATM. You need to prepend your call to the sproc with a DB specific command.

Author a = db.Query<Author>("EXECUTE GetAuthorById @0", 1).First();

The EXECUTE statement is database specific. If you are using MySql you need to use CALL instead which kind of starts breaking the abstraction a bit. I wonder if we will see this support getting more solidified in future releases.

ServiceStack ORMLite

At the time of writing this I couldn't see any built in support for sprocs using ORMLite. Either I am being my typical clueless self or we would just have to do it by hand. Here is what I came up with,

Author author = dbFactory.Exec(dbCmd => {
    dbCmd.CommandType = CommandType.StoredProcedure;
    dbCmd.Parameters.Add(new SqlParameter("@id", 1));
    dbCmd.CommandText = "GetAuthorById";
    return dbCmd.ExecuteReader().ConvertTo<Author>();
});

It's not exactly the worst thing in the world and I guess you could wrap this stuff in an extension method but still I feel spoilt by the other microORMs so far and this was a slight disappointment. Perhaps someone could either confirm or deny sproc support in ORMLite?

Simple.Data

Simple.Data has lovely syntax for this. You just call it as yet another dynamic method on the database object and Simple.Data just infers it for you.

Author author = Simple.Data.Database.Open().GetAuthorById(1).First();

Yeah thats nice.

Slightly Shorter and Summing Up

So I didn't want this one to have too much waffle (bar the odd text adventure here and there). I was concerned that perhaps calling sprocs might be one of those requirements were the whole microORM stuff starts falling apart but no. They actually handle the whole thing in a nice, clean manner.

Simple.Data is certainly starting to look more appealing to me and the whole fact that it's not database specific (you've got NoSQL stores and even experimental things such as a Registry adapter - imagine this as part of Powershell).

Anyway some code snippets for this stuff are currently available in the project/series github repo

¹ Not a real game.

Baler v0.3 Released

Thu Jun 16 23:00:00 UTC 2011

I've just pushed a new version of Baler. It's current version of 0.3 but semantically speaking it should be v0.2.2 as I haven't changed the API. But hey we all make mistakes :(

What's New?

Thanks to a suggestion from edazdarevic this release reworks the way files are generated.

Originally if a bale wasn't flagged as generated (e.g. after an IIS reset) a new file with a random name would be generated. This mean't that during development a shed load of files would be generated as you changed some other file and restarted your server. This lead to an awful mess in your output folder. Now what happens if a bale isn't flagged as generated the contents of the file are processed and a hash is generated from this. This hash is then used as the filename for that file (appended with the correct extension). If that file exists then the contents haven't changed and are not written out to a new file saving that extra file system write.

In the future I'd like to adopt the approach of simply writing out references to the bales contents during DEBUG/development mode and only process the bales when in PRODUCTION mode. But for now I think this is a reasonable stop gap.

edazdarevic has some more ideas that I plan on trying to integrate but I need more time to think about how to approach it without bloating the code.

MicroORMs for .NET: Syntax - SELECTs

Wed Jun 15 23:00:00 UTC 2011

This post is the is part of series of posts covering the various microORMs in the .NET world. I intend to look at how each microORM meets certain data access and manipulation needs. The current series includes,

Obligatory Introduction Post

Syntax - SELECTs

Stored Procedures

Inserts, Updates & Deletes

The two main things I want to look at in this article are

Syntax of the microORM, specifically around,
Selecting single and multiple results

So the most obvious starting point, bar the boring introduction, is how you perform selects using these awesome thingies. To begin with lets get our playground set up. Lets create a table of authors, lets call it Authors⁰

CREATE TABLE Authors(
  Id          int          IDENTITY(1,1) NOT NULL,
  Username    varchar(50)                NOT NULL,
  FullName    varchar(250)               NOT NULL,
  CreatedDate datetime                   NOT NULL,

 CONSTRAINT PK_Authors PRIMARY KEY CLUSTERED (Id ASC)
)

And populate it with some (meaningless) data¹,

INSERT INTO Authors(Username, FullName, CreatedDate) VALUES ('0@example.com', 'James Red', GETDATE());
INSERT INTO Authors(Username, FullName, CreatedDate) VALUES ('1@example.com', 'Harry Black', GETDATE());
-- ...CROPPED FOR BREVITY...
INSERT INTO Authors(Username, FullName, CreatedDate) VALUES ('134@example.com', 'Chloe Green', GETDATE());

Finally we need to create the equivalent C# object that our results can map to. Again nothing too crazy here,

public class Author
{
    public int Id { get; set; }
    public string Username { get; set; }
    public string FullName { get; set; }
    public DateTime CreatedDate { get; set; }
}

So now the stage is set and it's time for our microORMs to play their part.

Selecting Single and Multiple Authors

So lets assume we want to do 2 things -

Select a single author with an Id of 1, and,
Select multiple authors whose username (in this case the users email address) ends with example.com.

Dapper

Dapper has a single way of querying your database and it offers this via an extension method on the standard SqlConnection class,

using (SqlConnection conn = new SqlConnection(ConnectionString))
{
    conn.Open();
    Author a = conn.Query<Author>(
        "SELECT * FROM Authors WHERE Id = @Id", 
        new { Id = 1 }).First();
}

As you can see I've just opened a normal connection using a ConnectionString property I pulled from my App.config's ConnectionString config block. The magic here lies in the Query<Author> method. This method accepts a parameterised SQL string and maps the result back into the object you specified. Ultra simple, ultra clean. There are a whole host of other arguments to the Query<T> method but it would fill this post up pretty fast.

Selecting multiple object simply requires me to change the query to meet the new requirements,

IEnumerable<Author> dapperA = conn.Query<Author>(
    "SELECT * FROM Authors WHERE Username LIKE @PartialUsername",
    new { PartialUsername = "%example.com" });

One of the features I really liked apart from the simplicity of the whole thing was the parameterised SQL. Many of the frameworks support this but most support it by index based assignment e.g. "WHERE Id=@0". Thats fine but Dappers use of an anonymous object makes the whole thing that little bit nicer. No need to refactor strings should your arguments move around (it has happened) and it all reads just that tiny bit better as well.

Dapper also offers a non-generic Query method that returns instances of dynamic objects but I'm still to be convinced that this is useful when pushing these models into and back out of views etc. Still it's nice to have for those times that creating another model would be overkill.

Massive

Massive is slightly different to the other microORMs in that it only deals with dynamic objects from queries. This means if you need concrete models you will have to map them either manually or use a tool to help (such as AutoMapper). Due to my schema (pluarl table names) I use Massive in a slightly different way to most people. Massive requires your models to extend Massive.DynamicModel so to keep things clean I tended to naturally create a model that represents my Authors table,

public class Authors : Massive.DynamicModel
{
    public Authors()
        : base("DefaultConnectionString")
    {
        PrimaryKeyField = "Id";
    }
}

This provides an entry point for configuring how Massive connects to your database. The DynamicModel has a constructor that allows us to specify a connection string name that Massive will lookup from the Web/App.config ConnectionStrings config section. Alternatively if you don't supply this Massive will grab the first one it finds. This class also allows us to specify the table name (in this case inferred from the class name) and properties such as the primary key of the table (as shown above).

Now we have our class defined we can start selecting from the database. To select a single object we can use the Single method.

Authors table = new Authors();
dynamic result = table.Single(1);
Author author = Authors.Map(result);

You'll notice I've made another step here and that is mapping the dynamic result to a concrete Author object. It's a simple mapping method added to the Authors object but could be made more generic fairly easiy,

public static Author Map(dynamic obj)
{
    return new Author
    {
        Id = obj.Id,
        CreatedDate = obj.CreatedDate,
        FullName = obj.FullName,
        Username = obj.Username
    };
}

Selecting multiple objects can be done in 2 ways using All which returns the entire table and then you filter or using the Query method (preferred unless you really want to return the entire table for refdata or something I guess)

var tbl = new Authors();
IList<dynamic> dyn = tbl.Query(
    "SELECT * FROM Authors WHERE Username LIKE @0", 
    "%example.com");

UPDATE: Rob, the creator of Massive, has just pushed a new type of query syntax into Massive. The new approach makes use of named parameters

dynamic db = new Products();
var products = db.FindBy(
    CategoryID:8,
    UnitPrice:100
);

I haven't rewritten my stuff in this post but I want to close this post off and move on. There is plenty more info in the original post

PetaPoco

PetaPoco has a number of way of selecting single objects, or more specifically multiple ways of building/manipulating SQL queries. But before we do that we need to get access to the database. We do this via the PetaPoco.Database object which is an IDisposable wrapper around a Database connection that gives us the PetaPoco magic.

using (PetaPoco.Database db = new PetaPoco.Database("DefaultConnectionString"))
{
    // query goes here
}

UPDATE: Brad Robinson has informed me that the using statement is unnecessary

You don't need to explicitly dispose the database object unless you've called OpenSharedConnection yourself. The idea is that if you're using an IOC container to instantiate per-http-context instances you can call OpenSharedConnection at the start of the request and a single connection will be used for the entire http request. In practice I've found this not really necessary as most providers do connection pooling anyway. In other words, generally the using clause is not necessary.

The Database constructor takes a name of a defined connection string to use for connecting to the database. Now we are ready to start writing our query,

Using Single(id)

Author a = db.Single<Author>(1);

Using Custom Parameterised SQL

Author b = db.Single<Author>("SELECT * FROM Authors WHERE Id = @0", 1);

UPDATE: Brad Robinson pointed out that it is also possible to used named parameters as discussed in the Dapper section above @MyNamedParam instead of @0. The niceness just keeps coming with PetaPoco

Using SQL Builder to Append Statements

Author c = db.Single<Author>(
    PetaPoco.Sql.Builder
        .Append("SELECT * FROM Authors")
        .Append("WHERE Id = @0", 1));

Using SQL Builders Fluent Syntax to Append Statements

Author d = db.Single<Author>(
    PetaPoco.Sql.Builder
        .Select("*")
        .From("Authors")
        .Where("Id = @0", 1));

See what I mean? Lots of ways. Now I guess the 3rd and 4th methods will probably seem a bit OTT for this simple select but think how powerful these would be if we had to conditionally build up our where clause or select statement based on a number of user defined conditions. Nice.

Selecting multiple objects offers the same sort of variety except we can use either Query or Fetch. The difference between the two is summed up by the PetaPoco site,

These are pretty much identical except Fetch returns a List<> of POCO's whereas Query uses yield return to iterate over the results without loading the whole set into memory.

So what I say about Query applies to Fetch as well. Retrieving multiple objects is done in the same way as selecting a single object in that you can give the method parameterised SQL or use the SQL builder to construct the query.

IEnumerable<Author> authors = db.Query<Author>(
    "SELECT * FROM Authors WHERE Username LIKE @0", 
    "%example.com");

So to cover this off a full select using PetaPoco will would look like this.

using (PetaPoco.Database db = new PetaPoco.Database("DefaultConnectionString"))
{
    Author q = db.Single<Author>("SELECT * FROM Authors WHERE Id = @0", 1);
}

IMO The syntax is nice, clean and flexible. One thing about PetaPoco that is a VERY minor pain point for me is that it assumes singular name for tables which contradicted my earlier decision. Not to worry PetaPoco uses attributes against the model class to manipulate the standard configuration and I can specify a class level attribute of [PetaPoco.TableName("Authors")] to put things right with the world. This also only applies to the queries that use object inference such as Single<T> and not to the methods that accept SQL.

ServiceStack ORMLite

ORMLite from ServiceStack does things a wee bit differently to the rest. It's a bit less self-contained because it also applies extension methods to the string object as well. Also like PetaPoco it takes the singular naming convention for Object Name → Table Name so first things first we need to annotate our Author model with another attribtue

[ServiceStack.DataAnnotations.Alias("Authors")]

Another thing we need to do once before hand is set the dialect the ORMLite will use. This is so it knows how to speak to the specific database type we created. Typically you would put this in something like Global.asax.cs, Program.cs or whatever bootstrapper you care to use as it needs done only once.

OrmLiteConfig.DialectProvider = new SqlServerOrmLiteDialectProvider();

Now we can go ahead and start selecting things like a boss,

using (IDbConnection db = "...".OpenDbConnection())
using (IDbCommand cmd = db.CreateCommand())
{
    Author rows = cmd.GetById<Author>(1);
}

And for selecting many

using (IDbConnection db = Program.ConnectionString.OpenDbConnection())
using (IDbCommand cmd = db.CreateCommand())
{
    List<Author> rows = cmd.Select<Author>(
        "Username LIKE {0}", 
        "%example.com");
}

Alternatively if we want to perform a specific operation on each result we can use the Each<T> to iterate over the result set rather than simply return it.

As you can see you don't need to specify a full SQL query, just the WHERE clause which makes everything look a little bit neater and avoids repeating unnecessary portions of SQL that could be inferred.

ORMLite's syntax is slighlty more complex - it feels a bit closer to the metal. But it's certainly not horribly different that it would put me off using it.

UPDATE: It seems there is another way to query the database that makes things a bit shorter. Thanks to mythz (in the comments) for this one.

removed by original author :(

I agree that this is a viable alternative and can make for shorter code. Thanks.

Simple.Data

Finally Simple.Data offers a very clean syntax that mixes dynamic with concrete results. The first thing you need to do with Simple.Data is get a handle on the Database object that is returned as a special dynamic object. There are a number of ways of doing this,

Simple.Data.Database.Open();
Simple.Data.Database.OpenNamedConnection("DefaultConnectionString");
Simple.Data.Database.OpenConnection(Program.ConnectionString);

By using Database.Open() Simple.Data will look for a connection string called Simple.Data.Properties.Settings.DefaultConnectionString and use it to create the database instance. Alternatively you can specify a specific named connection using Database.OpenNamedConnection or even a raw connection string using Database.OpenConnection

These methods all return a special dynamic object that dynamically creates a SQL statement by constructing methods on the fly. It's best to look at the code for this. To select a single object we can do this,

Author author = Simple.Data.Database.Open().Author.FindById(1);

So whats happening here. Well firstly we a asking it to look at the Authors table. Next we are telling it to FindById - but this method doesn't exist. We could easily have said FindByUsernameAndEmail and we would get the results we expected. This bit of metaprogramming is the magic behind Simple.Data where method signatures are parsed and used to build SQL. Nice stuff. I still have a bit of issue working with dynamic objects but in this case my models are concrete so it's really not an issue.

Selecting multiple objects is that same except instead of FindByXXX we use FindAllByXXX and as this query returns a SimpleQuery object we then need to perform a .Cast<Author> at the end to get back a list of authors.

An alternative and more flexible way of doing queries is using Find and FindAll. These methods let you specify more complex queries. To implement our LIKE comparison for our query above we need to use FindAll like so,

dynamic authors = Simple.Data.Database.Open().Authors;
IEnumerable<Author> authors = authors
    .FindAll(authors.Username.Like("%.com"))
    .Cast<Author>();

This also shows the use of the Cast<T> mentioned above. We can supply a variable number of arguments to Find and FindAll as well as using different comparison operators.

That Was a Long One²

So yeah not only have I touched on selecting single and multiple object at a high level I also had to add a bit of discussion around setting things up so I apologise for the length of this post. On the plus side discussing configuration alone for MyBatis and nHibernate would probably take the same length of time. Thats whats great about these microORMs 99% of the time you don't need all that set up for most projects. No need getting bogged down in configuration when it's not actually achieving anything.

In terms of the syntax - so far I am quite taken by both Simple.Data and PetaPoco but that could change when I start really pushing things - after all everyone can do selects quite easily.

In terms of dynamic results (e.g. from Massive) I'm still to be convinced on their usefulness without having to always map to concrete classes. I don't see how this helps refactoring (it looks like it makes it much worse) nor do I see how it works with most validation mechanisms. But I would love to be proven wrong on this please.

One thing I do know - it's going to take a lot more justification for someone to use a heavyweight ORM in my next project. While I not willing to declare the death of ORMs like some other people I am certainly leaning in that direction.

Source for this series is slowly evolving in a Github repo if you feel inspired to have a poke around.

⁰ We can argue about plural or singular data naming conventions some other time, or in the comments!!!

¹ I created a load of test data using the handy little tool AutoPoco

² "Thats what she said" <self:facepalm>

MicroORMs for .NET: (Obligatory) Introduction (Post)

Tue Jun 14 23:00:00 UTC 2011

This post is the is part of series of posts covering the various microORMs in the .NET world. I intend to look at how each microORM meets certain data access and manipulation needs. The current series includes,

Obligatory Introduction Post

Syntax - SELECTs

Stored Procedures

Inserts, Updates & Deletes

I've been playing, somewhat superficially, with a few MicroORMs in .NET for a while now. While I have yet to use any in a real project I wanted to do a bit of comparison series. It's a bit selfish I guess as I am actually writing this series as a means to get myself up to speed on these frameworks as well as inform others about perils I faced as a newcomer to them as and when I hit those pain points.

To be honest I've never been a fan of the Data Access layer in projects - Databases are fine I guess and the application layer is all good but that thin layer between the two just frustrates me. It's often needlessly over complicated, confusing, hacked together or just plain broken. Most of my experience is with iBatis/MyBatis and it "does the job" but its very fiddly and XML configuration becomes cumbersome for simple tasks. I've also tinkered with other things like Entity Framework and NHibernate but more often than not they just don't feel right.

This make microORMs sound appealing to me for the very same reasons. They give you just enough to get the job done, prefer convention over configuration and generally allow you to get right down to the metal and tweak things using SQL - easily the best DSL for working with databases. Sure they may lack many features but typically these are features you wouldn't really miss if taken away from you.

The MicroORMs

There are a number of ORMs currently available for .NET and I am going to look at a few of them in terms of features and syntax.¹

Dapper

Dapper was created by Sam Saffron from Stackoverflow. Rather than a library Dapper is a single file that you can drop into your project and applies some syntactic sugar to extend your IDbConnection interface.

Massive

Like Dapper, Massive is a single file rather than a library and offers a very similar feature set to Dapper except it works exclusively with the .NET4 dynamic stuff.

PetaPoco

PetaPoco came out of inspiration from Massive and has extended to support real objects as well as some very nice extra features such as T4 templating, configuration via model attributes and other nice stuff.

Simple.Data

Simple.Data is A lightweight, dynamic data access component that make use of convention and dynamic to allow users to access database tables via dynamically generated methods.

ORMLite

ServiceStack.OrmLite is a convention-based, configuration-free lightweight ORM that uses standard POCO classes and Data Annotation attributes to infer its table schema.

On with the show

So thats my toys all laid out and ready to be played with. Next step is to look at some of the typical operations for selecting and manipulating data from our database. I hope to have the first real part of this investigation up shortly (or before the decade is out at least).

¹ This list is not exhaustive there are a few specialised ones around as well which I may touch on later in the series.

Baler v0.2.1 Released and 2 New Extensions

Wed Jun 08 23:00:00 UTC 2011

Baler

I have just released a new version of Baler. Version 0.2.1 release adds a new feature for adding custom attributes to bales. The Attr function, available as an instance method to all bales, accepts an attribute name and a string value that will be applied to the rendered bale tag. The best way to demonstrate this is with an example.

CSS Media Attribute

When specifying link tags for CSS it is possible to scope that stylesheet to a specific meida type. For example you could have a stylesheet that modifies your page style for printing e.g.

<link rel="stylesheet" href="core.css" type="text/css" media="screen" /> 
<link rel="stylesheet" href="print.css" type="text/css" media="print" />

The new Attr method allows you to specify this new attribute like so,

Baler.Build("core.css").Attr("media", "screen").AsCss();
Baler.Build("print.css").Attr("media", "print").AsCss();

"No Value" Attributes

Some attributes, such as the async and defer on script tags do not require a value. Attr is capable of dealing with these type of attributes.

Baler.Build("script.js").Attr("async").AsJs()

Will produce

<script type="text/javascript" src="..." async></script>

I am sure there are other uses for it (setting id's or data-* attributes) as it's a fairly flexible little enhancement.

CSS Media Extensions

PM> Install-Package CodeSlice.Web.Baler.Extensions.CssMedia

The first extension to utilise this new Attr is the CssMediaExtensions addon. CssMediaExtensions provide methods for setting the media type that a stylesheet should render for such as screen or print

WithMedia()

The WithMedia Extension allows you to set the media attribute of the rendered CSS tag.

bale.Build(...).WithMedia("screen").AsCss();

AsCss()

AsCss overrides the existing AsCss method allowing you to pass a media attribute value directly into the render call. For example the above example could simply be written as,

bale.Build(...).AsCss("screen");

Named Bales Extension

PM> Install-Package CodeSlice.Web.Baler.Extensions.NamedBales

This one has been released for a while but I didn't get around to writing about it. TheNamed Bales extension provides a mechanism to declare a Bale definition up front and reuse the same bale throughout the application without having to redefine the contents over and over. For example we can define a bale with 3 scripts and name it

Baler.Build(
  "script1.js", 
  "script2.js", 
  "script3.js"
).NameAs("base")

This can returns the Bale itself so can be used inline or somewhere like Global.asax. Next we can render this bundle using AsJs later in the app like so,

NamedBales.Get("base").AsJs();

Currently this doesn?t short circuit Balers internal cache so a second cache check will be made using the internal hashing mechanism. This is either a good thing or a bad thing. But for now it?s a good thing! This does have the side effect of ensuring that any unnamed bales with the same signature as a named bale will still get cached bale.

NameAs()

NameAs adds some sugar to the IBale interface allowing us to apply a friendly name to a bale. There is currently no check to see if the bale name is already taken. Existing definitions will be overwritten.

NamedBales.Get()

Allows us to retrieve a bale based on a friendly name defined by the developer. Will throw and exception if the bale name doesn?t exist in the cache ? otherwise the bale itself.

Coming Up

Still plenty to be done. Docs are fairly complete but there are a mountain of unit tests to be written still and some extra features to be added (without bloating the code base of course). I have a few features I'd like to work in but I'll keep that for another day.

Baler Site now Live(ish)

Sun Jun 05 23:00:00 UTC 2011

I've just pushed the first draft of the new shiny Baler web site into the various tubes that form the basis of the interwebs. I am still missing a logo but why let that hold me back? Anyway feel free to check it out while I add more tests to the project and generally polish things.

Baler Project Site

Announcing Baler

Mon May 30 23:00:00 UTC 2011

It's very early days (with regards to both code and documentation) but I though I amy as well put it out there.

Baler is a .NET Web Resource Bundler. Capable of concatenating and transforming CSS and JavaScript files to lower request count and bandwidth requirements. It works with any .NET Web Framework/View Engine capable of running C# in the view

What differentiates Baler from other resource bundlers out there is that it aims to provided a bare bones, minimum feature set for building bales (or resource bundle). The core Baler package offers the ability to concatenate, render and cache JavaScript and CSS bales. Thats it. No minification nor fancy processing nothing. However Baler does have 2 main extensibility hooks though (Before and After) which can be leveraged to control how bundles are manipulated. There are currently 3 extensions for Baler but more are on their way,

CoffeeScript - Transforms CoffeeScript files into JavaScript
.LESS - Transforms .LESS files into CSS
Ajax Minifier - Allows for minification of JavaScript and CSS using Microsofts Ajax Minifier

Quick Example

<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="Default.aspx.cs" Inherits="CodeSlice.Web.Test.Default" %>
<%@ Import Namespace="CodeSlice.Web.Baler" %>

<!DOCTYPE HTML />
<html>
  <head>
    <%=
      Baler.Build(
        "~/scripts/script1.js",
        "~/scripts/script2.js",
        "~/scripts/script3.js",
      ).AsJs()
    %>
  </head>
  <body>
  </body>
</html>

This example will take 3 source files concatenate them into a random script file and output the necessary <script src='1jdk2ds.js' type='text/javascript'></script>. Subsequent calls that match this bales signature (the order and contents of the bale) will use the cached script tag and not perform that concatenation again.

As I say it's early days. I really need to get the docs up to snuff, write plenty of test and add a few more features before I start shouting from the rooftops but at least it's out now and there is no going back :-).

Baler v0.1 is available on Nuget and Github and the annotated source code is available here, here, here and here

Turns Out I'm Not Doing my Job Very Well aka Sucks To Be Me

Wed May 25 23:00:00 UTC 2011

Soooooo anyway I had the day off today and you know what they say - the devil makes work for idle hands. And boy did that devil dude really go to town on me. Earlier this morning I came across a tweet. The sort of tweet that I like. A highly polarising tweet that makes geeks stand up and cheer and makes everyone else feel like we are laughing at them. Anyway heres the tweet,

"pragmaticpat: adults are "pro homework" for children, yet frequently do not study their craft outside of m-f, 9-5 #leadbyexample ?"

So anyway I thought, as I was waiting in the car for the wife to finishing shopping or whatever she was doing at the time, I would throw this tweet up to our Yammer community and stir up a bit of debate. Things got a bit heated, more than even I would have liked. In fact I think I offended people¹.

Lets say just I am beyond passionate about my job almost obsessively so in some respects. Being that type of person I often expect everyone else to be just like me and therefore measure people against what I do. I also tend to make statements as if there were 2 types of people in the world. People like me, who go home and do a bit of extra work/research and people who do their 9-5 allocated work and don't care about anything. But hey there's a twist here - I'm not actually stupid - I know the world isn't black and white, us and them, all or nothing. I'm also not as horribly snobby or elitist as I make out (I am a bit but thats just the nature of a technology fanboy - defensive about their passions - suck it).

Anyways enough of the woe is me schtick, it's boring. I just want to say that I understand there are people out there who do their 9-5 and go home and actually have a life beyond the keyboard. I was never implying people need to work 24/7 or they are coasting. I realise I have a bit of a reputation of being an elitist uber-geek and therefore many people may have assumed that was what I was getting at. However I know that in the 9-5 block a lot of people are contributing more that just their time to a single project and are going out of their way to do something decent rather than mediocre. I'm happy for someone to say "I'm happy with were I am" - hell I am sure they've worked hard to get were they are so why not? What I'm not happy with is someone who expect the same kudos as people who go above and beyond just because they show their face everyday. When I retweeted that tweet and even went as far as yamming it I was stating an opinion, an opinion that wasn't directed at any particular people or any particular company for that matter, it was just a blanket statement.

On a similar note I've had the pleasure of seeing plenty of CV's over the last few years as well as interviewing people and I must say the industry has it's fair share of people who are guru's on paper but ask them a question about something that they haven't encountered due to work on a project and they are flummoxed. It's a sad state of affairs to get into that shape. It's a bit like a morbidly obese fitness instructor - they can do their job but neglected to look after themselves - who would want to hire that sort of person? Not many people I'd bet.

So yeah I accept that there are many different types of people. People who want to better the industry as well as themselves. People who want to better themselves and the company they work for. People who just want to better themselves. People who just want to do a super job with the role they are given and finally people who just want to get by². But I refuse to accept that everyone should be treated equally in all respects and as long as people can respect that then we can all get along.

So whats the point of all this? I know I have high standards but I also need to learn that not everyone wants the same thing out of life. Or more to the point that other people actually HAVE a life outside of IT. In the past I've had shed loads of opportunity to shine, but I've also been on some long boring pressured projects so I need to realise other people are going through that too. I need to be a bit smarter about this. It's even part of my job description to mentor others so I suck at my job (hence the title). Perhaps I should pull my finger out and ensure people are given opportunities to shine? Yeah thats sounds about right….

¹ For what it's worth I wasn't asked, request or ordered to write this. As always I am doing this for my own reasons.

² I'm sure you could break it down more.

Micro Web Frameworks in .NET - Nancy - Hello World

Tue May 24 00:00:00 UTC 2011

Part of my Micro Web Frameworks in .NET series. This time looking at Hello World in Nancy nancyfx.org.

Micro Web Frameworks in .NET - Jessica - Views

Mon May 23 00:00:00 UTC 2011

The fourth part of a series covering the various micro web frameworks in .NET. This part shows you how to serve dynamic views with Jessica http://jessicafx.org.

Micro Web Frameworks in .NET 101: Nancy

Sun May 22 23:00:00 UTC 2011

Nancy is a different beast to Jessica while on the surface it may appear like a micro web framework, once you scratch the surface it becomes debatable because there is just some much more than a nice DSL for creating lightweight web apps. In fact this was one of the reasons Jessica was created,

Nancy has taken an approach to making many parts of the framework replaceable; if you don?t like the functionality of X you can create your own, and have Nancy use that instead. While this is a respectable goal for a web framework, the focus for Jessica has been to stay as simple and as close to Sinatra as possible. Tom Bell. Why Did I Create Jessica?

Anyway we will touch on these aspects a bit later. Lets get cracking.

Getting Started - Hello World

Getting Nancy up and running is simple enough. Starting with an empty ASP.NET web project¹ we can use NuGet to install Nancy

Install-Package Nancy

Also because we are hosting this in an ASP.NET environment (more on this later) we need to install the appropriate hosting package

Install-Package Nancy.Hosting.Aspnet

Next step requires us to add some stuff to the web.config. If you've been smart and installed the hosting package from NuGet this steps is done for you but ultimately your minimal web.config should look like this.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <system.web>
    <compilation debug="true" targetFramework="4.0" />
    <httpHandlers>
      <add verb="*" type="Nancy.Hosting.Aspnet.NancyHttpRequestHandler" path="*" />
    </httpHandlers>
  </system.web>
  <system.webServer>
    <validation validateIntegratedModeConfiguration="false" />
    <handlers>
      <add name="Nancy" verb="*" type="Nancy.Hosting.Aspnet.NancyHttpRequestHandler" path="*" />
    </handlers>
  </system.webServer>
</configuration>

The final step is to create the hello world module that will handle your route. Again, like Jessica, it doesn't matter where the module classes are placed but for neatness I just like to stick them in a Modules folder. Our module will look very similar to the one we defined for Jessica.

public class HelloWorldModule : NancyModule
{
    public HelloWorldModule()
    {
        Get["/"] = r => "Hello World";
    }
}

And we're done. Fire the app up and behold the "Hello World"-yness of it all.

Other Micro Framework Related Stuff

Nancy has some features that aren't technically related to the micro framework aspect - specifically hosting but I'll touch on that later. Nancy also has a lot of nice features that are related to the micro framework aspects so lets take a bit of a whirlwind tour around the main areas first.

Routes

Routes in Nancy are pretty powerful. Obviously you get the declare static routes (as demonstrated above) but dynamic routes (routes with variables for example) are also supported in a few of ways. First of all you have named segments. These can be expressed in 2 ways

Sinatra style variables /user/:id, or,
C#-esque String.format style /user/{id}

Declaring a route like this will cause the variable section to be pushed into a named variable within the dynamic object passedin into the route action. /user/jameshu matches /user/:id and the dynamic object passed into the action will have x.id == "jameshu"

The other means of dynamic route matching is the use of regular expressions and named capture groups (or backreferences as they are sometimes known). Using regular expressions as route parameters allows us to offer more fine grained route matching. Take the following route as an example,

Get[@"/users/(?<id>[\d]+)"] = r => GetUser(r.id);

This route will ONLY match routes whose variable section matches the expression, in this case a numeric value. So not only have we specified a route variable but we are also limiting access to that action to "valid" values. This means we could also offer another route that accepts only alphabetical characters. Powerful enough.

Route Conditions

Taking routes another step further Nancy also offers optional conditions that can be used to determine if a route should be executed for a given request. Lets use this heavily contrived example to demonstrate this feature,

Get["/user", ctx => ctx.Request.Query["password"] == "secrets"] = r =>  "Hello World";

So this route will match any route starting with /user however the condition (the second argument of the definition) will ensure that the route action will only get executed if the query string is populated with an argument called password with a value of secrets i.e http://myserver/user?password=secrets will match successfully.

Views

Nancy supports a decent range of view engines²

Razor
Spark
NDjango and, obviously,
Static files

Serving up dynamic vies is fairly simple

public class HelloWorldModule : NancyModule
{
    public HelloWorldModule()
    {
        Get["/"] = r => View["index.cshtml", "Hello World"];
    }
}

We basically use the View object to specify the view and optionally pass in a view model as well. By default Nancy will look in the ~/Views folder for the views but you can also specify a full path if needs be. The view is then matched against the available view engines and rendered as you would expect. This is the Razor view specified above.

<!DOCTYPE html>
<html>
    <head>
        <title>Hello World</title>
    </head>
    <body>
        <h1>@Model</h1>
    </body>
</html>

Responses

Nancy is also capable of returning static content as different responses (similar to how Jessica does it). Each route action is expected to return a Nancy Response object. The object itself has a number of convenience methods for serving static files and JSON such as Response.AsJs, Response.AsJson, Response.AsXml, , Response.AsRedirect etc.

Hosting

This is were things move away from "micro framework" world a bit. Nancy itself abstracts out the hosting environment making the entire framework portable allowing you to host in any number of places including (but not limited to),

Self hosted (e.g. embedded within an executable)
ASP.NET/IIS (as demonstrated above)
WCF (yep thats right), and,
Some partial OWIN support

This, to me at least, is very interesting. Being able to build an executable that accepts HTTP requests could be pretty powerful in the right hands and the WCF stuff sounds very interesting (is this similar to what the new WCF Web API is attempting to offer?). I need to play some more in this area. I'll report back when I am done.

Summing Up

So that was a quick round up of the key features of Nancy and I think it's another one worth looking at. So hopefully you can also see how it differs from Jessica. From a code perspective they seem quite similar (though Nancy has possibly a few more features) but the whole hosting abstraction layer is very interesting. I have had people ask me if this introduced any new (or even old) security concerns and to be honest I don't know but it might certainly be worth looking out for.

As always I may need corrections so please fire away. Hopefully I can cover off these points with a few more screencasts over the next week. Stay tuned.

¹ In my Jessica I laid out steps to strip the normal ASP.NET Web Project template out of all the unnecessary bits. Lord knows why I didn't just create an empty web project <facepalm/>

² Apparently it also has a built in view engine called the Super Simple View Engine but as of yet I can't seem to get it to work.

Micro Web Frameworks in .NET Screencast: Nancy #1 - Hello World

Sun May 22 23:00:00 UTC 2011

Nancy #1 - Hello World from James Hughes on Vimeo.

Part of my Micro Web Frameworks in .NET series. This time looking at Hello World in Nancy (http://nancyfx.org)

Micro Web Frameworks in .NET - Jessica - Configuration

Sun May 22 00:00:00 UTC 2011

The third part of a series covering the various micro web frameworks in .NET. This part shows you how to configure Jessica http://jessicafx.org.

Micro Web Frameworks in .NET - Jessica - Static Files

Sat May 21 00:00:00 UTC 2011

The second part of a series covering the various micro web frameworks in .NET. This part shows you how to serve static files through Jessica http://jessicafx.org.

Micro Web Frameworks in .NET - Jessica - Hello World

Fri May 20 00:00:00 UTC 2011

The first part of a series covering the various micro web frameworks in .NET. This part shows you how to implement Hello World using Jessica http://jessicafx.org.

Screencasts Galore! More or less.

Thu May 19 23:00:00 UTC 2011

SCREEEEEEEEEEEEEEEEEEEEEEENCASTS GALORE!

So I unveiled the new screencasts section of the site. Can you guess whats in it? Those who said "screencasts" you can have a nice non-patronising pat on the head. The rest of you…. for shame. Tut tut.

Anyways I wanted to start a series of short screencasts that will provide a practical view of the .NET Micro Web Framework series. So far I've managed to cover the basics of Jessica but there is more to come.

Micro Web Frameworks in .NET Screencast: Jessica #4 - Views

Thu May 19 23:00:00 UTC 2011

Jessica #4 - Views from James Hughes on Vimeo.

The four part of a series covering the various micro web frameworks in .NET. This part shows you how to serve dynamic views with Jessica (http://jessicafx.org).

Micro Web Frameworks in .NET Screencast: Jessica #3 - Configuration

Thu May 19 23:00:00 UTC 2011

Jessica #3 - Configuration from James Hughes on Vimeo.

The third part of a series covering the various micro web frameworks in .NET. This part shows you how to configure Jessica (http://jessicafx.org).

Micro Web Frameworks in .NET Screencast: Jessica #2 - Static Content

Thu May 19 23:00:00 UTC 2011

Jessica #2 - Static Content from James Hughes on Vimeo.

The second part of a series covering the various micro web frameworks in .NET. This part shows you how to serve static files through Jessica (http://jessicafx.org).

Micro Web Frameworks in .NET Screencast - Jessica #1 - Hello World

Thu May 19 23:00:00 UTC 2011

Jessica #1 - Hello World from James Hughes on Vimeo.

The first part of a series covering the various micro web frameworks in .NET. This part shows you how to implement Hello World using Jessica (http://jessicafx.org).

Micro Web Frameworks in .NET 101: Jessica

Wed May 18 23:00:00 UTC 2011

Now I have got the opinion piece out of the way I can start the bit I like - dissecting the various technologies that could be classed as micro web frameworks. First on the table is Jessica.

Getting Started - Hello World

Getting a simple Hello World set up in Jessica involves (optionally) taking more out than you put in. Jessica requires very little configuration so creating a new web project, deleting all the extra files and folders, trimming the web.config and emptying the Global.asax.cs makes for a nice blank canvas to start with.

Next step is pretty difficult :-). Within the NuGet Package Manager Console just install the Jessica package

Install-Package Jessica

Once installed be need to tell Jess to initialise/bootstrap itself when the application starts by calling the initialise method in the Application_Start method of the Global.asax.cs

public class Global : System.Web.HttpApplication
{
    void Application_Start(object sender, EventArgs e)
    {
        Jess.Initialise();
    }
}

At this point it's also possible to supply a number of configuration options but we will touch on this later along with the optional configuration XML. Now that Jess is setup to configure itself all we need to do now is define the module that will handle our complex Hello World output.

Modules form the basis of Jessica each one holding a number of routes (based on URL's and HTTP verbs) and associated actions. Our app will have a single module,

public class MainModule : JessModule
{
    public MainModule()
    {
        Get("/", r => "Hello World");
    }
}

This simple module defines a single route that will match any standard GET request for the root of the app. And that's it. When you run the app Jess will inspect the assembly and find all the modules and wire up the routes.

The Rest of the Pie

Obviously serving plain bits of text is a bit useless so there are plenty of things that Jessica offers that makes it, well, actually useful.

Configuration

Jessica can be configured programatically or via XML config. A number of things you can currently set include,

Environment - the environment you build in e.g. development, test, release. Default development
Public Directory - The directory to be used to server static content (html, js, css, images) that require no preprocessing. Default public
Views Directory - The directory to be used to serve views that can be dynamically rendered (Razor, Textile, Markdown, Liquid etc.). Default views

The XML config is outlined below but each of these properties can be set in the Global.asax.cs programatically as well via the Jess.Configuration properties.

<configSections>
    <section name="jessica" 
             type="Jessica.Configuration.JessicaConfiguration,Jessica"/>
</configSections>
    
<jessica environment="development" 
         publicDir="Public" 
         viewsDir="Views"/>

Routes and Actions

Routes may be defined as a literal path such as /users or /store/toys or with variable segments such as /user/:id or /blog/:year/:month/:day. Variable segments are then parsed out of the route url and the values added to the action argument.

Get("/blog/:year/:month/:day", r => 
{
    var blog = Blog.Get(r.year, r.month, r.day);
    return View("blog", blog);
});

That seems to be about the extent of the route manipulation for now. Would be nice to see the other Sinatra stuff included such as splats, regular expressions and conditions.

Actions are fairly simple they take a dynamic object and are expected to return a Jessica Response object. The dynamic object passed in contains a number of things such as route parameters, querystring parameters and the current HttpContext. As an added niceness the Jessica Response object has a number of implicit type conversions that allow us to return simpler objects

public MainModule()
{
    // string -> text/html
    Get("/", r => "Hello World");

    // int -> http status code
    Get("/", r => 404);                             

    // Action<Response> -> Response stream
    Get("/", r => Response.AsText("Test")); 
}

Static Content

Jessica allows us to return static content simply. First of all all routes are first checked for the existence of a file at that URL. If a file is found it is served, if not then Jessica goes to the route table and works it's magic. Putting resources in the publicDir allows Jessica to serve static content from the route actions and the Response object comes with a number of nice helpers to help make your static actions neater.

public MainModule()
{
    Get("/", r => Response.AsCss("test.css")); 
    Get("/", r => Response.AsFile("trojanz.exe", "application/application"));
    Get("/", r => Response.AsHtml("test.html"));
    Get("/", r => Response.AsJs("test.js"));
    Get("/", r => Response.AsJson(new { Name = "James", Age = 31 }));
    Get("/", r => Response.AsRedirect("http://www.google.com"));
    Get("/", r => Response.AsText("test"));
}

Views

What web technology would be complete without the ability to serve non-static views? Oddly enough Jessica, out of the box at least, comes with no view engines. I suspect this is because dynamic view rendering isn't technically something that should form the basis of a web framework - thats the job of the view engines. Jessica does however come with an extension point of adding X amount of view engines - Jess.ViewEngines. There is also a healthy volume of view engines available for Jessica,

Rendering a view from an action is also very simple

public class MainModule : JessModule
{
    public MainModule()
    {
        Get("/user/", r => View("index"));
        Get("/user/:username", r => View("user", r.username));
    }
}

The first option renders a simple view called index while the second takes a model object that is pushed into the scope of the view and can be used to build the view. Jessica will loop through it's ViewEngines collection and attempt to resolve the view name to a file in the specified viewsDir folder. If it finds a matching view it will render it.

I am going to take the lead from Jessica itself and not dive into the various view choices for now. They are pretty self explanatory anyway.

Summing Up

So thats the 101, the skinny, the lowdown, the starter for 10, the basics and other random statements of Jessica. Hopefully this covers the vast majority of what v0.83 of Jessica offers. It's a nice minimla framework that attempts to stick to Sinatra's philosophy. It would be nice to see a few more of Sinatras features creeping in especially route conditions. There is also some nice high level error handling being introduced (for v0.9 I assume) for handling application wide errors and 404's. I plan on talking about these when they make an appearance.

Corrections, comments and criticisms below please.

Why Would You Use Micro Web Frameworks in .NET?

Tue May 17 23:00:00 UTC 2011

UPDATE: Oh dear James <shakeshead/>. So it seems someone was paying attention to my post and noticed my benchmarks were hideously flawed. I have removed them because I don't like looking like a total fool but I will be replacing the with honest, real and unstupified results as soon as I can. Apologies for any distress this may have caused. FUD is a powerful thing in the wrong hands but ignorance is worse.

I've been doing a bit of research into micro web frameworks in the.NET world recently and I was posed a question by a colleague.

Whats the point of a micro web framework?

I gave the usual response - more lightweight, smaller foot print blah blah blah. To be honest I was barely convincing myself. I suspected I had fallen into the trap of latching on to something because it was novel to me rather than any real reason. I am not saying they don't have a place in solutions technology stack, quite the opposite in fact, what I am saying is that I didn't have sufficient justification for bothering with them.

So I've spent a bit more time looking at the background of these microframeworks rather than the techs themselves and hopefully the next time someone asks I can give a more convincing reply.

What Is A Micro Web Framework

Pretty much a micro web framework is a framework that provides the absolute minimum feature set required to create a web application. Extra features can be added via extensions etc. but the sole purpose of a microframework is to allow the app to handle requests to certain URLs (routing) with a bit of basic error handling for 404's etc. (possibly). No feature creep, no bloated deployments just ultra-lean request/response handling.

The most famous microframework, and the one that started it all, comes from the Ruby world - Sinatra and it's "Hello World" example really highlights the ease of creating an app.

require 'sinatra'

get '/hi' do
  "Hello World!"
end

Run this and navigate to /hi to see the amazing response of Hello World.

Micro Web Frameworks in the .NET World

So how does this compare to similar technologies in the .NET world. Most .NET solutions take a lot of inspiration from Sinatra (and why not it's pretty much the blueprints for the microframeworks) and this one is written using Jessica.

public class MyModule : Jessica.JessModule
{
  public MyModule()
  {
    Get("/hi", p => "Hello, world!");
  }
}

OK so there is a bit of extra language noise but this is to be expected when moving from Ruby to C# but it's just as readable. Jessica isn't the only microframework in the .NET world but it does seem to be one of the better ones (based on a very superficial amount of research - no offence meant to other frameworks - happy to be proven wrong). The other frameworks considered "micro" include,

What Makes Micro Frameworks Worthwhile?

So lets expand on my original argument of "more lightweight, smaller foot print blah blah blah". The most obvious is that a microframework by design is going to be a lot more lightweight than a normal framework. It's lifecycle for a request is going to be greatly simplified and ultimately this should help increase performance per request (I currently have no figures to back this up yet but I am working on it). As well as performance and increased throughput this lightweight approach should greatly reduce the memory footprint which will help your app scale a bit better¹.

Another reason to consider a microframework is that you are in control of everything and that is good in my book. It's one of my arguments for choosing MVC over WebForms. No unnecessary injection of client side code or script managers or any of that sort of thing. No overly complex view properties being populated for no reason. Nothing but the stuff you put in.

Not exactly within the bounds of a microframework by definition but still worth mentioning is hosting. Nancy has a number of hosting options

Standalone
In a web app
In a WCF Service

This sort of swappable hosting greatly increases portability of your app. Now this is Nancy specific but the effort required to create this sort of functionality with another microframework is trivial in comparison to the what it would be in MVC or WebForms.

Also many of the microframeworks have some sort of support for OWIN which is essentially a web abstraction that would allow your app to run on any OWIN supported server³. This sort of support would take longer to roll out across the larger frameworks which generally stick to IIS. I'm not saying it won't happen but the bigger your framework the slower change is going to happen.

That brings me to my final point - evolution. ASP.NET MVC has evolved very fast since v1.0 and there is no sign of that progress abating any time soon. Still it's speed is hindered by it's size. These microframeworks introduce new features (some experimental for the brave among us) much faster because of the fact they are small. Extending these frameworks don't require the creation of numerous classes to do simple things - there is generally a common extension point with a very small set of requirements.

Conclusion

I hope this has helped my colleagues get a better picture of why I think microframeworks are worthy of our time and why I am keen to do a bit of research on them. I also hope this has broadened a few minds in terms of solution architectures in the .NET world. Not everything needs to be some configurable, Enterprise ready, heavyweight application sometimes the simplest solution is all you need.

¹ Provided you don't do something stupid!

² As a friend pointed out it's easy to write a fast Hello World server so take these results with a healthy does of skepticism

³ It's early days for OWIN but still worth mentioning.

Further Adventures in Unit Testing Technologies

Mon May 16 23:00:00 UTC 2011

This post is a follow up to A Tale of Unit Testing Technologies. Information about the project repository can be found on my project page.

I've finally managed to slog through the long and slightly tedious process of re-writing the same suite of tests in a number of C# and JavaScript unit testing technologies - the results of which can be seen on the project GitHub page. My conclusion, especially in the .NET world, is that it doesn't really matter that much. I know it's not that much of a brilliant ending but what did you expect? Fireworks?

Anyways before I start amazing you all with more awe inspiring revelations let me run through the remaining technologies (until such times as more are added at least) and summarise their good and bad points. The other technologies I brought on board include,

Some obvious ones there just to round out the comparisons and a few JavaScript based ones.

NUnit

Good old NUnit. Whats not to love? Mark a class as a fixture and mark your methods as a test.

[TestFixture]
public class TagTest
{
    [Test]
    public void TagNameNormalisationTest()
    {
        Model.Tag tag = new Model.Tag {  Name = "My Tag Name" };
        Assert.AreEqual(tag.NormalisedName, "mytagname");
    }
}

No point in mentioning too much about NUnit as most people will know the ins and outs of it already. It has VS integration (via TestDriven) and a nice NUnit runner capable of watching changes to DLL's and automatically rerunning the test contained within. One caveat around that is often NUnit hangs on to the DLL reference and requires a restart so Visual Studio can rebuild the DLL which kind of misses the point - but it doesn't happen too often.

The framework itself would lend itself to a bit of betterification through sugar such as Should and Should.Fluent but it does what it says on the tin.

MSTest

Another staple for many people. I'd never really used it too much prior to doing this post because of the general negative opinion of it. I must admit I kind of agree with some of these things. It's pretty much the MS equivalent of NUnit (in terms of features and look and feel)

[TestClass]
public class TagTest
{
    [TestMethod]
    public void TagNameNormalisationTest()
    {
        Model.Tag tag = new Model.Tag { Name = "My Tag Name" };
        Assert.AreEqual(tag.NormalisedName, "mytagname");
    }
}

So why do so many people prefer NUnit over MSTest? Well I think the main problem is that it is quite slow in comparison (from Visual Studio at least) but my main gripe is the amount of flob it generates for each test run and configuration. It even produces solution folders for holding test runs and other files. If I want you to create those I'll ask you - don't force it on me. It also has a very limited set of Assertion methods much like NUnit.

Enough moaning - surely it must have some benefits? Well yes it does of course. The main benefit IMHO is that it has very nice integration into Visual Studio which offers some very clean feedback. This is something that requires TestDriven for NUNit and the other frameworks which may or may not be free depending on your situation.

NSpec

Not to be confused with the first hit on Google which seems to be pretty much a dead project NSpec (.org) is an attempt to bring RSpec into the .NET world.

class describe_Tag : nspec
{
    void when_setting_the_tag_name_to_My_Tag_Name()
    {
        before = () => _tag = new Tag { Name = "My Tag Name" };
        it["should normalise the tag name to mytagname"] = () => 
            _tag.NormalisedName.should_be("mytagname");
    }

    private Tag _tag;
}

This is somewhat similar to MSpec I mentioned in the last post. My bugbear with MSpec however was that it didn't feel natural to me. The equivalent MSpec test for the above looks like this,

[Subject("Normalise Tag Name")]
public class when_a_tag_is_created
{
    static Tag _tag;

    Establish context = () =>
        _tag = new Tag();

    Because of = () =>
        _tag.Name = "New Test Tag";

    It should_have_a_normalised_name_of_newtesttag = () =>
        _tag.NormalisedName.ShouldEqual("newtesttag");      
}

Maybe it's just me but it doesn't read as well as the NSpec version. I even found it easier to write the NSpec version - more natural.

NSpec also has a lot of extra goodies. Currently there is no integration into Visual Studio yet but the runner is simple to enough to spin up from the NuGet Package Manager Console and it has an added bonus of a file watcher. SpecWatchr is similar to NUnit it can watch for changes and automatically run the specs. It differs from NUnit in that it watches changes to the code files rather than the DLL - so need to wait for builds to happen etc. Nice.

One minor gripe with it (bar the lack of VS support - even TestDriven support would be good) is the should syntax. I prefer the Should.Fluent style syntax Should.Be() rather than the current should_be syntax. It's not Ruby and we should accept that :-P.

It's still quite early for NSpec and hopefully it keeps it's momentum as it would be one of my frameworks of choice.

Quixote

Quixote by Rob Conery follows on from his current slew of ultra lightweight solutions to common problems (like Massive and Sugar). Quixote blends the HTML reporting directly with the framework and you write tests right in a Razor file. This is the definition of lightweight.

@using Quixote;
@using CodeSlice.UnitTesting.Model;

<link href='@Url.Content("~/Styles/quixote.css")' rel="stylesheet" type="text/css" />

@TheFollowing.Describes("Tags")
    @They.Should("Normalise the Tag Name", () => {
        return new Tag { Name = "Test Name" }.NormalisedName.ShouldEqual("testname");
    })

The obvious problem with this is that you don't get continuous integration or build support. There is no need to rebuild anything just refresh your page and the tests re-run. Still CI/Build integration would be nice for a framework.

Jasmine

Ah Jasmine. Jasmine is what I wanted out of a JavaScript testing framework. Rather than going the JSpec route of creating a preprocessed DSL Jasmine makes use of JavaScripts dynamic nature and "functions as 1st class citizens" feature to create framework that behaves like RSpec but embraces JavaScript.

describe('Tag', function() {
    var tag;

    beforeEach(function() {
        tag = new Tag();
    });

    it("should normalise a tag name", function(){
        tag.setName('A Tag Name');
        expect(tag.getNormalisedName()).toEqual('atagname');
    });
});

Out of the box you get an extensible framework (custom matchers etc), mocking and asynchronous support. Ajax faking isn't directly available but I recommend using Sinon for this as it has an amazing FakeServer object that is fully configurable.

Another big advantage of Jasmine is it's build integration. Be it node.js, Java or Ruby you can integrate Jasmine specs into your build process. .NET integration isn't available yet but Jurassic would be a suitable host if anyone wants to make it happen (hint, hint :-P).

QUnit

And finally QUnit. QUnit is the NUnit/JUnit equivalent for the JavaScript world. Well known and mature project with some very nice HTML output.

module('Tag Model Validation');

test('Tag name normalisation', function(){
    var tag = new Tag({ name: 'A Tag Name' }),
        normalisedName = tag.getNormalisedName();

    equals('atagname', normalisedName);
});

Not much else to say on this one right now.... must be running out of steam.

Summing Up

And there you have it. The first real post since the move and it's a bit of a waffly long one. Anyways what would be my recommendations? Well for .NET I'm torn. Currently I'd say NUnit with Should.Fluent extensions. This is a nice combo for creating simple tests that people can execute from within Visual Studio. However NSpec has potential - I'd like to see it grow a bit more and I'll keep watching it. Visual Studio integration would be aces though. JavaScript on the other hand I have a clear cut winner. Jasmine is awesome. It's boosted my code quality and productivity no end on recent projects and with the help of Sinon for mocking and controlling Ajax it'll be a long time before I am this happy with a .NET equivalent tech. Fingers crossed for NSpec on that front.

The First Day of the Rest of My Life

Sun May 15 23:00:00 UTC 2011

Move complete, more or less. Still a few broken images and stuff but like the unpacked boxes of a new house these will probably remain that way until about next year or so. Decent enough weekend project all in. For anyone who's interested in what's went on under the hood - read on.

Abandoning Posterous

Posterous is great as a simple blogging platform but in terms of giving me exactly what I wanted at any point in time Posterous and most other platforms didn't give me exactly what I wanted (I am quite a demanding character I guess).

I was doing some research on .NET Micro Web Frameworks and came across Jessica which was created and maintained by Tom Bell.

Jekyll

Browsing Toms github repo I discovered he prodced his site using Jekyll a Ruby based static site generator and I knew I had to have it! So I ripped off Toms nice design, uglified it with my own tweaks and what you see now is the finished site. This site is also mirroed on Github but some of the functions (category pages) are missing due to some custom generators that the Github Jekyll instance doesn't run.

There is more to be deployed but I'd rather polish what I have before pushing more stuff out. I've tinkered with Project page generation, Showoff for presentations and a few other things that will appear over time.

Domain Name

Seriously - how couldn't I go for a shiny web 2.0 ultra elite domain name given the chance? Gandi.net was recommended to me and I must admit the set up was fairly painless and their admin panel stuff is very nice.

Appharbor

Currently my static site is hosted on appharbor. Appharbor is awesome. Typically I've used it to spin up quick demos and stuff and the simplicity of doing a git push to deploy your new version is like a dream (especially given the heavy handed approach of Azure).

Thats It

Not sure that anyone cares about this sort of stuff but I just want to take my new process for a spin anyway. Any questions - let me know.

On The Move

Fri May 13 23:00:00 UTC 2011

So I've decided to go ahead and move away from Posterous - the call of Jekyll was just too much. I've migrated a lot of my content but there is some dodgy formatting and plenty of missing images so I am in the process of sorting everything out.

Also thanks to Tom Bell for letting me use his template.

Providing a Foundation for .NET Projects using NuGet

Thu May 05 23:00:00 UTC 2011

It's always good to have a foundation, right? I mean if you have a heap of projects going on or you work for a company that has a load of active or future projects then why not have a foundation? Foundations are good. I mean they promote DRY cross projects, reduce development time for many common tasks (logging, data access, etc.). Foundations can also provide a number of tools and technologies that are “approved” — that is they have been throughly researched and you aren't going to trip up over license issues, inexperience of the tech or shortcomings of the tech/tools.

The problem with foundations is that people start assuming they are some sort of magical framework that is supposed to do all the work. You wrong son! A foundation works best when it's a pile of techs/lightweight solutions to common problems. They don't work too well when you just get a big monolithic DLL that acts as an abstraction for other technologies. Abstractions are good for certain things but not everything. A big dll is also harder to document. How do you know whats in it bar using Intellisense or reflector? OK OK OK there should also be some sort of associated documentation that comes with it but keeping these two things inline is extra work. Extra work that doesn't get done — it's not funded work so why bother? Eh?

In our company we use this approach and it works from a technology perspective but from a people perspective it could be better. There is a general lack awareness of what is contained within the magical 3 dlls that get generated and as such less stuff get pushed into the foundation as it probably could. The current process works like so,

Create a new Subversion repository for the project
Branch the current Foundation (a Visual Studio Solution) repository for the current project
Tweak the build step of the Foundation solution to dump it's dlls into the lib folder of the project
Commit foundation changes to the project foundation branch
Review and selectively merge changes into the trunk of the Foundation repository

Even writing the steps out and thinking about all the undocumented steps required to get this to work smoothly tells me that it is possibly more complicated than it has to be. In fact the merging back into the Foundation trunk doesn't happen as much as it should. In fact projects adding thing to their own branch of the foundation rarely happens cause it's a bit messy.

Another issue with this approach is that it only makes sense for bespoke code. I mean if you are building out a dll how do you tell people that “actually JSON.NET is our preferred JSON Serialiser”. It's not easy as the documenting and announcing of such things is in another document out in the ether.

An Alternative Approach

Modular. It's the future I tell thee! Discrete pieces of functionality that can run independently. This enables project to take the parts of the foundation they want and build a framework from it. Much like ender (http://ender.no.de/) is doing in the JavaScript world currently being able to take micro-libraries and build a macro-framework that suits your needs and just that. No risk of featureitis, evolve your framework as you need and be in complete control doing it.

Thats the plan but how do we achieve it? HINT: The answer, a partial one anyway, is in the title of this post. Well, first things first we would need to take our current foundation and modularise it. Along with that we'd probably need to ensure tests etc. are up to scratch (start on the right foot). Secondly we would need to change the delivery mechanism - no more branching and merging of codebases - too heavyweight. NuGet (nuget.codeplex.com) will help. With NuGet we could create packages from our modules, manage dependencies, document usage right in the package manifest, and expose these packages as a server and searchable package repository (a la nuget.org). Into this repository we could push all our approved technologies (researched and vetted previously) as well as new versions of our bespoke code. As time goes on this repository can grow and more people can add to it. Heck we can even make the submission process as easy as we see on nuget.org. Wunderbar!

So how would this affect how projects operate on a daily basis?

Well the mechanism for the typical usage is greatly simplified. The project pushes and pulls packages from a central package repo. People use the repo to search for things they need (JSON Serialiser, Micro-ORM etc) and if they don't find something they can then push a new package into the repo (or just suggest one). Self documenting use of 3rd party techs, all available all searchable.

Another advantage of this approach, and using NuGet in particular, allows us to bundle not just dlls but also transform files for configuration (web.xml etc.), javascript/css/other client side assets, project and solution wide tools (executable that can be run from the VS command prompt etc.) and even code samples and templates. This gives us a richer, more cohesive delivery mechanism for a foundation to a project.

Of course it's not all perfect. Urgent changes to bespoke code will need to be done the old fashioned way and pushed to the main server which is less than ideal but it's a bit of an edge case.

So whats not to like about this idea? Foundations not frameworks is the key here. Let the project create its own framework as that will work much better than imposing one upon them.

So this is a bit of a brain dump (some would say brain fart no doubt) as usual and I am open to suggestions and corrections. I've likely missed a few glaring problems with the new approach so let the critics loose and I'll try my best to address their concerns.

Next step - setting up a simple POC to test the general workflow.

Flashbag: Glimpse

Wed Apr 20 23:00:00 UTC 2011

Glimpse is an extensible client side debugging dev tool for server side stuff in any ASP.NET (WebForms) and ASP.NET MVC (RoR and PHP forthcoming). Think Firebug for your server and you have the gist of it. This presentation is a very brief introduction/overview/review.

Also the quality totally blows I promise it'll be better next time.

Flashbag (fuh-lash-bah-ag): A flashbag is a group of people who assemble suddenly in a meeting room type place, partake in a brief technical discussion (or demonstration), then disperse, often for the purposes of education and betterization

Guerrilla Learning Experiment #2: FlashBags

Wed Apr 20 23:00:00 UTC 2011

So I kicked off another experiment in my war against ignorance in the IT industry. To make myself look cool I am using the term Guerrilla Learning. The experiment? Well I am calling it FlashBag*

Flashbag (fuh-lash-bah-ag): A flashbag is a group of people who assemble suddenly in a meeting room type place, partake in a brief technical discussion (or demonstration), then disperse, often for the purposes of education and betterization.

The name itself is a portmanteau of Flash Mob (I assume you all know what this is by now) and Whitebag - a technology awareness session held during lunch - hence Whitebag (the one you bring your lunch in).

Anyway whats it all about then? Well a typical scenario would be this.....

I am reading my tech blogs on a day and come across a new tech that seems really nice.
As usual I link it on yammer or twitter or whatever and continue to dive into it.
I'm sold. This is something other people need to know about.
So I book a room for 1/2 hour at the last minute, post internally the room and the topic (say 12:00, Boardroom, Glimpse for .NET)
Once the party is assembled I do a very short talk, possibly demo (15 minutes tops) and thats it.

So what would this achieve? Well I believe a lot of people throw links and stuff into a “Read Later” pile or have a quick look at the page and go “Cool”. Then they forget all about it. This quick interactive, community focused practical sessions forces them to actually think about whats going on (James Hughes PHd in Theoretical Theories and Things like that) and how they would apply it to their work. It's more in your face and creates a better connection in those peoples minds. The news then precipitates further - the people attending the FlashBag post on yammer/twitter and talk about it more rather than just little old me.

As always I record these things as a memento of my awesome presentation skills (#lies) and allowing other people to see what went on.

It's early days and I've done one so far (posting shortly) - turn out was mediocre but perhaps that will change, perhaps it won't. Mediocre turn out is better than none I guess - perhaps people are busy, perhaps enthusiasm is waining among my colleagues, or perhaps they are sick of me spamming every sense they have on a daily basis!

Thoughts?

* After some internal discussion the name is up for debate. I like it but mostly because of my huge ego. Thoughts?

Crucible Code Review Guidelines

Mon Apr 18 23:00:00 UTC 2011

I've been using Crucible on a number of projects for the last few months and I love it. So now comes the time we want to try and make a case for it to be rolled out to other projects in the company I work for. Along with the usual stuff such as cost of licenses, logistics, hardware need and so on I also need to consider education and best practise guidelines (oh I said “best practise” that term makes me feel slightly queasy) because after all — if people aren't using it right they probably wont see the benefit, stop using it and then we are back to square one with wasted time, money and effort.

Some of you may be thinking,

“But it's just a code review tool - what guidelines do you really need?”

But truth be told you do need some, and I learnt this the hard(ish) way. So here are some top tips to getting the most out of Crucible and at the same time possibly embettering (not a real word but should be) your development approach.

Rule 1: Review Early, Review Often

Don't wait for a whole week before committing 600 files as a single review — no ones going to actually do a proper review as they don't have time or patience. Generally speaking your work should be sufficiently broken down so that you can do a few hours of work, commit that code and submit that changeset as a review. Typically speaking a small review may only take about 5 minutes or so to review per person which is a lot easier than having to actually book time to read an entire books worth of code. Obviously don't be going insane on this. A code review should encompass a complete feature and not some arbitrary files that are only a small piece of the puzzle. Feel free to merge changesets where necessary too.

Rule 2: Refrain from adding new content to old reviews

I've seen a few cases were a bunch of files have been added to a review and this review has been used over and over with people adding files and new versions until the whole thing had pretty much the entire codebase included. Needless to say that's wrong. Create a review and don't be adding new files or versions too it unless it's in the early stages of review and it is deemed necessary. Don't be afraid of kicking off more than one review - it's not a crime, in fact it's advised

Rule 3: Don't be TOO iterative

Another problem I've experienced is that a review can run for a very long time. This is linked to Rule 2 in that, based on my review comments, people will make the changes and bring that file in that review up to date. This leads to further comments and the whole thing just consumes too much time. Crucible gives you the power to do this but you need a bit of discipline to simply know when to stop and when to start a new review.

In recent projects I have been performing a review, making comments etc. and completing the review (unless there is anything serious I want to make sure gets addressed ASAP). When changes are made they generally get picked up in the next set of reviews anyway and as I talk to my team all the time I can give a brief sanity review when the changes are made.

Making your submitted reviews lightweight helps with this.

Rule 4: Keep reviews small

Again similar to Rule 1 and 2. Don't be submitting hundreds of files for a review. If you need to make a big review considering breaking it down into smaller feature sets and describing this in the review. It makes it easier for reviewers to perform the review and for yourself to make the changes quickly (if any)

Rule 5: Let everyone get involved

Gone are the days when a single (supposedly) smart person performs all reviews. Social code reviews are and should be the future. Embrace this. Every single dev on my projects are added to reviews by default, even people with little to no experience of the technology. Not only does it help people spot things that others may miss but it also

generates debate and discussion around coding style
helps people get an understanding of the coding practise in the project
acts as a learning tool for people
helps people feel more involved in the project and team

Conclusion (cause I can't think of a better word)

Hopefully these guidelines give you a better understanding of how I work and how I get maximum benefit from Crucible. After all it is just a tool and therefore open to abuse. Honestly it's better than our previous approach (literally using Word documents) and I can't imagine anyone would prefer our old approach to this. Does anyone else have any recommendations around the use of Crucible?

PowerMockup â Wireframe Toolkit for PowerPoint

Fri Apr 15 23:00:00 UTC 2011

Link: http://www.powermockup.com/

Disclosure Time: I approached the developer of this app offering to do a review in exchange for a review license. This isn't the sort of thing I usually blog about (except internally, a few times, in the company I work for) but I'm all for supporting the people who deserve support and truth be told I would probably have written this blog post anyway (yep it'll be a positive one)

Happy with that? Good lets crack on.

The Past

I've written a fair number UI design documents over the last few years and for all of these I have used Balsamiq. The problem with that is our companies standard/approved wire-framing solution is Visio (with the GUUUI templates). Why did I go against the grain? Well I am not a mad fan of Visio. It's not exactly lightweight, in fact it's extremely bloated (at least in the wire-framing sense), and that is easily the worst thing you can ask for when dealing with highly iterative, dynamic, interactive wire-framing. You need to work with the Visio rather than let Visio work for you and that is slightly too much fail for me most of the time. Admittedly there has been times I prefer the slicker, more professional, output of Visio (our customers come from the stiff collar world of finance and the like so it tends to sit better with them). But were I can I stick with Balsamiq. Until now....

The Present

The problem I've always faced with Balsamiq is that when I produce a first draft of a document with, say, 20 odd wireframes in it and something fundamental is changed by the customer then I have a hard slog ahead to fix all those images. Thankfully a colleague of mine, one much better at this wire-framing/UX stuff than me, pointed me towards PowerMockup. Rather than make a new standalone application that has it's own random UI features and offers little to on integration into the app we use day today PowerMockup is simply an extension/add-on to Powerpoint. So what does this mean? Well it means,

The devs spent more time into the templates and stuff that matter
Fully integrated into the Office application stack (think ability to tweak results when embedded into Word)
Already makes use of the built in “smart” styling and other features that make Powerpoint actually a fairly decent app for quick templating and stuff. In fact take it a step further and you could make use of Powerpoint's animation stuff to demonstrate UI interactions.

So given 5 minutes what could I do? And I do mean 5 minutes (possibly slightly less). Well I mocked up the website you are reading right now.

Trippy eh? It's not my best work (far from it) but it did help me test drive PowerMockup and get a better feel of it. So I like it. I like it because of all the things I've mentioned above (who wouldn't). I'd use it, and will use it, as my wire-framing tool of choice (and not because I got a free license - my work would pay for it anyway cause they are total aces) from now on but it's not perfect. If I had to pick holes there are a few things that I'd mention,

My biggest peeve but it's not a show stopper is the location of the various shapes/widgets. Dropdowns are in “Navigation” and Text fields are in “Text” but 99% of the time I'd want both of those when constructing a form. I'd really like a “Form” collection with all of these things grouped.
If I drop a browser container on the page then drop a scrollbar on the page I need to resize said scrollbar to fit the browser container. Problem is, as the scrollbar is a grouped object, it will stretch the entire thing rather than extend the base rectangle. Again no show stopper but it is a bit of a pain. I guess offering the components as separate shapes might make it easier (just a suggestion).
There a plenty of icons but I'd like to see a few more (“comments” icon, “tags” icon) - the equivalent famfamfam icon set would blow me away (and also a lot to ask). Possibly some more mobile shapes etc. would also be nice.

The Future

So thats it for now I am very happy, not because I managed to blag a free license, but rather because the app/plugin/whatever fits into how I do things so seamlessly. Good work and I look forward to future updates - hopefully more people think the same.

Technically Non-Technical: The Pomodoro Technique

Tue Apr 05 23:00:00 UTC 2011

I've been going through a period recently of attempted self improvement. Trying new things to make myself a better more effective code monkey — iterative code reviews, todo lists, time management, pair programming, TDD, BDD, la dee dee....

My old strategy was simple — get given or discover a piece of work, usually pretty large, and start coding. Once done do a bit of exploratory testing, read the spec to check I've covered everything, commit the code and submit a bit of a code review. These sessions tend to last a number of days which leads to an eventual short term burnout. Now because I always write 100% flawless code first attempt that's generally fine but it doesn't really allow other people to learn from my excellence. Also being the smartest person on any project I am generally being pulled left and right answering queries and explaining my very clever designs — this creates regular context shifts and makes it harder for me to get my focus back on creating all the awesome!

One of the techniques I have found quite successful recently was the Pomodoro Technique. The name comes from the original kitchen timer used by the creator and the focus is squarely on a low tech approach. The premise behind it is simple but effective. You focus on your task for a 25 minute period (a pomodoro) ignoring all external interruptions (or you forfeit the pom), take 5 minute break, repeat. After 4 pomodoros you take a longer break (15-30 minutes). The rules are as follows,

A Pomodoro Consists of 25 minutes Plus a Five-Minute Break.
After Every Four Pomodoros Comes a 15-30 Minute Break.
The Pomodoro Is Indivisible. There are no half or quarter Pomodoros.
If a Pomodoro Begins, It Has to Ring:
- If a Pomodoro is interrupted definitively – i.e. the interruption isn't handled it's considered void, never begun, and it can't be recorded with an X.
- If an activity is completed once a Pomodoro has already begun, continue reviewing the same activity until the Pomodoro rings.
Protect the Pomodoro. Inform effectively, negotiate quickly to reschedule the interruption, call back the person who interrupted you as agreed.
If It Lasts More Than 5-7 Pomodoros, Break It Down. Complex
activities should be divided into several activities.
If It Lasts Less Than One Pomodoro, Add It Up. Simple tasks can be combined.

My Honest Opinion

My previous development strategy was all kinds of broken. It worked provided I wasn't disturbed and I didn't really have a point at which I could turn round to someone and say

“give me 5 minutes and I'll get back to you”

The pom gives me that excuse and with the use of a simple widget I am able to know when I'll next be free (and so are other people). The hardest thing for me to accept was actually taking the 5 minute break post-pom. Initially (and sometimes still) I'd just work over the 5 minutes — just one more line of code, just test it once more — excuses, excuses, excuses. Breaking old habits is hard! Some of the other benefits from the technique is that,

it really helps you visualise your task up front and break it down much better
it helps you track work that you have done and how long tasks take.
some pomodoro software even integrates into popular task lists and calendars

Resources

The Pomodoro Technique website is by far the best resource for this stuff and even has a free ebook on the technique and its background

The site: http://www.pomodorotechnique.com/
The book: http://www.pomodorotechnique.com/products.html#pomodoropdf

Personally it's going to take me while longer to get into the right mentality 100% but I think the benefits will be worth it. One day I'll be so efficient the code will just stream out of my brain and into my IDE - but for now I'll stick the mortals way of doing things.

A Tale of Unit Testing Technologies

Sun Apr 03 23:00:00 UTC 2011

I've had a big bad dirty secret for a while now. One I've kept hidden from my friend and colleagues for years. But now its time to come clean...... here goes..... hold onto your pants.

<pause for dramatic effect>
I hardly ever write unit tests!
<gasps>

I've come clean and it feels good. Another reason why it feels good is that it is slowly but surely becoming a falsehood. I've had my eyes opened recently to the art of BDD and discovered that I've simply been doing it wrong - I was so busy writing code first and then tests to fit my, possibly incorrect, assumptions that tests failed to hold any worthwhile value — time wasted in my mind. Also when writing said tests I was too worried on the internal mechanics of the "unit" being tested and not about its intended behaviour — essentially missing the point and time wasted once more. Being a man that has no time for, well, time wasting I feel I have come full circle on why I didn't test. In the past at least....

But I digress.... Where was I? Ah yes I've had my eyes opened - I've been doing it right. Yep I've gone all TDD/BDD - write a failing test, write some code to make it work, make the test fail again and so on and so forth. Its been a good experience. I finally see the benefit in it. Yes, it takes slightly longer and it's hard to break old habits but the amount of times I've been tripped up by assumptions and edge cases that would only normally be found during system testing after a frustrating debugging session with plenty of hair pulling and swearing.

Since the whole behaviour driven enlightenment I've been trying to find a suitable set of technologies that allow me to perfect my new found approach and turn me into a fast and effective code cutting machine. NUnit is fine — it does the job but it's a task in itself to express what you want out of a test which slows things down. No if you're going to do it right you want the best tools. So I have decided to jump in head first and discover what testing tools are the most effective. I've been through a few already , including,

SpecFlow (C#)
MSpec (Machine.Specifications) (C#)
JSpec (JavaScript)
Should/Should.Fluent (C#)

Here are a few of my thoughts so far.

SpecFlow (http://www.specflow.org/)

SpecFlow is essentially the .NET equivalent of Cucumber (from the Ruby world). Offering natural language syntax for defining scenarios that make up the behaviour of a feature. Its better as an example,

Feature: Tags

Scenario: Normalise Tag Name
	Given I have created a new tag
	When I set its name to "New Test Tag"
	Then it should have a normalised name of "newtesttag"

So you see the test here is written in pretty much straight english. It's easy to understand and this means even domain experts (aka "the business folk") can help write them. So how does this end up being turned into executable tests. Well, initially you need to do a bit of wiring up using Step Definitions.

[Binding]
public class TagSteps
{
    Model.Tag _tag;

    [Given(@"I have created a new tag")]
    public void GivenIHaveCreatedANewTag()
    {
        _tag = new Model.Tag();    
    }

    [When(@"I set its name to ""(.*)""")]
    public void WhenISetItsNameToNewTestTag(string name)
    {
        _tag.Name = name;
    }

    [Then(@"it should have a normalised name of ""(.*)""")]
    public void ThenItShouldHaveANormalisedNameOfNewtestag(string normalisedname)
    {
        _tag.NormalisedName.Should().Equal(normalisedname);
    }
}

As you can see we can match up each line of the feature file with an appropriate executable action. We even have the power of regular expressions at our disposal - allowing us to reuse steps and build up a decent library that comes with us across projects. Some very very powerful stuff here — we can get everyone involved in writing these things,

Developers and business users during design
Developers during development
Testers during system test — in fact why not write a feature as the "Steps to reproduce"? Win, win!

The thing is...... well.... Technically speaking SpecFlow isn't really aimed at the fine grained, unit test level of testing. It's more about the higher level integration testing. I mean it's useable but it does become a bit awkward to express things in a reusable manner. The problem with that is things start feeling quite heavy weight and you need to do a lot of extra work to get them to fit. No I think, while SpecFlow is great for many things (automated testing, integration testing, system testing etc.) it's not the best fit for what I am looking for in this article - unit testing tools.

MSpec (https://github.com/machine/machine.specifications)

Machine.Specifications (MSpec for short) is a Context/Specification framework geared towards removing language noise and simplifying tests.

Thats the official intention and I must admit I was initially taken by it. Rather than having a single huge class filled with methods representing tests MSpec takes the approach that a single class represents a single scenario and uses lambda expressions to offer the BDD style syntax (Because/It/Subject etc.). So lets take the Tag scenario described above and convert it to MSpec format,

[Subject("Normalise Tag Name")]
public class when_a_tag_is_created
{
    static Tag _tag;

    Establish context = () =>
        _tag = new Tag();

    Because of = () =>
        _tag.Name = "New Test Tag";

    It should_have_a_normalised_name_of_newtesttag = () =>
        _tag.NormalisedName.ShouldEqual("newtesttag");      
}

It's quite simple and it really forces you to ensure your tests are as simple as possible. To be honest having used it on a project I'm not 100% sold. I think its down to my personal coding style. I honestly think while it reduces line count it doesn't necessarily reduce language noise. Also having used on a project I found it quite awkward to write — the style is different to what I am used to and I guess with training that would change. As I say this is a personal opinion and I am happy to be shown a better approach.

JSpec (http://visionmedia.github.com/jspec/)

JSpec is a clever little JavaScript testing framework. I's no longer supported by it's creators (booooo!) but honestly its stable enough to use anyway. JSpec takes the RSpec DSL (a decent subset at least) and is capable of converting it to JavaScript for execution in the browser. It's got a heap of stuff in it,

Mock Ajax
Stubbing
Fake timers

Lets take a look at a simple example,

describe "Utils.Arrays.compare method"    
    it "should compare and sort 2 numeric arrays successfully"
        arr1 = [1,2,3]
        arr2 = [3,1,2]
                
        Utils.Arrays.compare(arr1, arr2, true).should.be true        
    end            
    
    it "should compare 2 numeric arrays unsuccesfully when not sorted"
        arr1 = [1,2,3]
        arr2 = [3,1,2]
                
        Utils.Arrays.compare(arr1, arr2).should.be false        
        Utils.Arrays.compare(arr1, arr2, false).should.be false        
    end        
end

People who have used RSpec before will feel right at home. People who haven't should be able to understand exactly what is going on. Pretty - right? I thought so. For the people who think — "ugh we don't need another dialect/language" - wise up! Out of all the testing techs I've used recently this one has been the most successful and the output it generates is nice and clean.

Should/Should.Fluent (http://should.codeplex.com/)

This little gem isn't a framework in itself and can be used with any framework you care to use. Should provides a more expressive way of stating assertions in your code by making the code closer to natural language using extensions methods and nicer method names. Example I hear you say? Why certainly sirs and madams,

public void Should_fluent_assertions()
{
    object obj = null;
    obj.Should().Be.Null();

    obj = new object();
    obj.Should().Be.OfType(typeof(object));
    obj.Should().Equal(obj);
    obj.Should().Not.Be.Null();
    obj.Should().Not.Be.SameAs(new object());
    obj.Should().Not.Be.OfType<string>();
    obj.Should().Not.Equal("foo");

    obj = "x";
    obj.Should().Not.Be.InRange("y", "z");
    obj.Should().Be.InRange("a", "z");
    obj.Should().Be.SameAs("x");

    "This String".Should().Contain("This");
    "This String".Should().Not.Be.Empty();
    "This String".Should().Not.Contain("foobar");

    false.Should().Be.False();
    true.Should().Be.True();

    var list = new List<object>();
    list.Should().Count.Zero();
    list.Should().Not.Contain.Item(new object());

    var item = new object();
    list.Add(item);
    list.Should().Not.Be.Empty();
    list.Should().Contain.Item(item);
};

I stole this one from the Should Codeplex site (linked above) and it makes use of the fluent syntax (optional). OK technically it doesn't do much but it really helps when trying to express assertions in your tests. Highly recommend this one.

Conclusion

Thats the first lot of technologies covered but there are still plenty out there. I think my next port of call is to actually spin up IronRuby and get RSpec involved. After all it is the marker by which I am comparing these things so why did I not jump on it first of all? Perhaps I like to build up suspense :-P

Also worth pointing out is that I haven't touched upon technologies for mocking and stubbing - that is for another time.

As always heap criticism my way and I'll happily fight my corner and stubbornly refuse to back down :-P (second smilie within a few paragraphs time to end this post).

UPDATE: I've pushed some of my code to GitHub and intend to expand on this using the various other technologies. The repository can be found on my repo on GitHub

Serialising .NET Resources to JSON for Web Apps

Mon Mar 28 23:00:00 UTC 2011

Pop quiz hotshot - you are building one of them massive single page web apps using something like Sencha where the vast majority of work is pushed into the JavaScript realm and some crazy nut wants everything localised or at least all text strings push to RESX files. What do you do? WHAT DO YOU DO?

Option 1

Well the most obvious option, and IMHO the most horrible, would be to convert all those JavaScript files into aspx's or cshtml's or whatever and embed the resource references directly into the files.

<%@ Import Namespace="Some.Resources" %>

var myWidget = new Widget({
    title:       '<%=Strings.DefaultWidgetTitle%>',
    description: '<%=Strings.DefaultWidgetDescription%>'
});

myWidget.show(document.body);

This is far from nice. For one it kind of makes any static compression of the files either impossible or at best annoyingly fiddly. It also prevents any sort of quick client side caching unless you use some sort of VaryByCulture Output Caching strategy on the server side. Blegh.

Option 2

The next option, and certainly much better would be to pull out the direct <%= %> references and store them in another smaller file and reference them through a global JS object

<%-- DECALRED WITHIN THE MAIN HTML PAGE --%>
<script type="text/javascript">
    var Strings = {
        DefaultWidgetTitle:       '<%=Strings.DefaultWidgetTitle%>',
        DefaultWidgetDescription: '<%=Strings.DefaultWidgetDescription%>'
    }
</script>

var myWidget = new Widget({
    title:       Strings.DefaultWidgetTitle,
    description: Strings.DefaultWidgetDescription
});

myWidget.show(document.body);

This is nicer because it means you can compress and cache the JavaScript file globally without having to worry about different cultures etc. It does leave one annoyance though. The Strings object above it essentially boiler plate. The names are a 1:1 mapping of the resource file so we have introduced a layer of abstraction we have to write manually leaving us open to make some mistakes. Slightly less blegh, but still blegh.

Option 3

The solution I like the best is to provide a mechanism for serialising the Resource file into the equivalent JSON object. You get all the benefits of option 2 while not having to worry about having to write the mapping file. So lets keep the JS file from the second option and change the ASPX file

<%-- DECALRED WITHIN THE MAIN HTML PAGE --%>
<script type="text/javascript">
    var Strings = <%= ResourceSerialiser.ToJson(typeof(Some.Resource.Strings)) %>
</script>

Now lets look at the magic behind this option - the JSON Serialiser

/// <summary>
/// Utility class that allows serialisation of .NET resource files (.resx) 
/// into different formats
/// </summary>
public static class ResourceSerialiser
{
    #region JSON Serialisation
    /// <summary>
    /// Converts a resrouce type into an equivalent JSON object using the 
    /// current Culture
    /// </summary>
    /// <param name="resource">The resoruce type to serialise</param>
    /// <returns>A JSON string representation of the resource</returns>
    public static string ToJson(Type resource)
    {
        CultureInfo culture = CultureInfo.CurrentCulture;
        return ToJson(resource, culture);
    }

    /// <summary>
    /// Converts a resrouce type into an equivalent JSON object using the 
    /// culture derived from the language code passed in
    /// </summary>
    /// <param name="resource">The resoruce type to serialise</param>
    /// <param name="languageCode">The language code to derive the culture</param>
    /// <returns>A JSON string representation of the resource</returns>
    public static string ToJson(Type resource, string languageCode)
    {
        CultureInfo culture = CultureInfo.GetCultureInfo(languageCode);
        return ToJson(resource, culture);
    }

    /// <summary>
    /// Converts a resrouce type into an equivalent JSON object
    /// </summary>
    /// <param name="resource">The resoruce type to serialise</param>
    /// <param name="culture">The culture to retrieve</param>
    /// <returns>A JSON string representation of the resource</returns>
    public static string ToJson(Type resource, CultureInfo culture)
    {
        Dictionary<string, string> dictionary = ResourceToDictionary(resource, culture);
        return JsonConvert.SerializeObject(dictionary);
    }
    #endregion

    /// <summary>
    /// Converts a resrouce type into a dictionary type while localising 
    /// the strings using the passed in culture
    /// </summary>
    /// <param name="resource">The resoruce type to serialise</param>
    /// <param name="culture">The culture to retrieve</param>
    /// <returns>A dictionary representation of the resource</returns>
    private static Dictionary<string, string> ResourceToDictionary(Type resource, CultureInfo culture)
    {
        ResourceManager rm = new ResourceManager(resource);
        PropertyInfo[] pis = resource.GetProperties(BindingFlags.Public | BindingFlags.Static);
        IEnumerable<KeyValuePair<string, string>> values =
            from pi in pis
            where pi.PropertyType == typeof(string)
            select new KeyValuePair<string, string>(
                pi.Name,
                rm.GetString(pi.Name, culture));
        Dictionary<string, string> dictionary = values.ToDictionary(k => k.Key, v => v.Value);

        return dictionary;
    }
}

Simple enough little class that is configurable by Culture etc. so you can pull different translation out on demand if needs be. Obviously it doesn't do anything around caching - SRP and all that stuff you know :-P

Yeah so I've used this on 2 projects already with great success so hopefully someone else finds it useful.

CoffeeScript: JavaScript sans Bad Parts

Sun Mar 13 00:00:00 UTC 2011

This was written a long time ago when coffeescript was in early beta and I've decided to push it into the public domain. I've updated it a bit but I apologise for any oversights.
Also worth mentioning that I am a lot more taken with it than I was at the time of the original blog. I will be revisiting this again soon.

If I've said it once, I've said it a million times - JavaScript is misunderstood. Sure it's got it's bad parts (eval, ==, typed wrappers etc.) and it's got it's VERY bad parts (global variables, scope, typeof etc) but it's also got a lot of beautiful parts (functions as first class objects, object and array literals, dynamic objects and prototypal inheritance etc.). JSLint validates your code against the good subset to ensure you aren't using all that nasty stuff that is going to break your code eventually but can we take it a step further? What if we could take out the good subset and create a subset of the language so we couldn't ever use the bad stuff? Well turns out you can.

Enter CoffeeScript

CoffeeScript is a full featured subset of JavaScript. Essentially it is a meta-language that compiles into JavaScript. Its syntax is a bit different, more like Perl, Python or Ruby and the JavaScript it generates only uses the "good parts" of JavaScript and passes all strict JSLint tests.

Syntax

CoffeeScripts syntax is close to JavaScript and besides a few extra goodies it is nearly a 1:1 mapping. However it strips away a lot of the boilerplate stuff to make the code you write slightly more succinct. Another feature of the syntax is that indentation is important (like Perl or Python, I can never remember which) as there are no line ending tokens. So lets look at a few statements

# Assignment:
number   = 42
opposite = true

//-- GENERATED JS 

// Assignment:
var number, opposite;
number = 42;
opposite = true;

This simple example of variable creation and assignment demonstrates how CoffeeScripts avoids one of the most common and potentially fatal mistakes made by many JavaScript developers - Inadvertent global variable declaration. In plain old JavaScript leaving out a var statement on a variable declaration results in the variable being declared (or overwritten -eek!) in the global scope. In CoffeeScript var is optional and all variable declarations are made in the current scope (and pushed to the top of the scope as is the JavaScript way). Want global scope declaration for some random reason? Use window.variable name and then ask yourself why you are doing it wrong.

# Conditions:
number = -42 if opposite

//-- GENERATED JS 

// Conditions:
var number;
if (opposite) {
  number = -42;
}

# Functions:
square = (x) -> x * x

//-- GENERATED JS 

// Functions:
var square;
square = function(x) {
  return x * x;
};

Function declarations are a good example of how succinct CoffeeScript can be in comparison to JavaScript. In fact anyone keen on C#'s lambdas or Groovys syntax will find this quite familiar. Gone are the bloating function and return keywords ( -> is the empty function - function() {})

# Objects:
math =
  root:   Math.sqrt
  square: square
  cube:   (x) -> x * square x 

//-- GENERATED JS 

// Objects:
var math;
math = {
  root: Math.sqrt,
  square: square,
  cube: function(x) {
    return x * square(x);
  }
};

# Existence:
alert "I knew it!" if elvis?

//-- GENERATED JS 

// Existence:
if (typeof elvis != "undefined" && elvis !== null) {
  alert("I knew it!");
}

Existence is a nice feature as it keeps non-boolean falsy values nice and strict and saves a lot of guessing. That's the basics but as I mentioned earlier there are some nice bonus features that can make some common tasks very clean.

The Goodie Bag

Array Comprehension

# Array Comprehension:
squares = ((x) -> x * x) n for n in [1,2,3]

//-- GENERATED JS 

// Array Comprehension:
var n, squares, _i, _len, _ref;
_ref = [1, 2, 3];
for (_i = 0, _len = _ref.length; _i < _len; _i++) {
  n = _ref[_i];
  squares = (function(x) {
    return x * x;
  })(n);
}

Array comprehension is quite a common task in many JavaScript solutions. Simply put this is the Map part of any typical MapReduce situations (common in many NoSQL style databases and Ajax situations). MapReduce is essentially - given an array of items,

Map: Transform that list into what you want (SQLs SELECT)
Reduce: Based on that mapping remove unwanted items

In the Ajax world you'd probably need to do this when dealing with JSON or XML responses which are quite often generally result sets. The example above shows just how easy it is to apply an arbitrary function over an array of items. Now I guess the equivalent JS could be written slightly better by hand but would it not take longer and be harder to maintain? Probably. This being quite a common task the less you have to write the better!

Splats

Splats are a convenient way to work with the arguments object in JavaScript while clearly outlining required and optional arguments. Lets take this example.

# Splats:
log = console?.log or (o) -> alert o
printResults = (winner, runners...) ->
  log "Winner:", winner
  if runners?
    ((r) -> log "Runner up", r) for r in runners

printResults "James", "Paul", "Martin", "John"

//-- GENERATED JS 

// Splats:
var log, printResults;
var __slice = Array.prototype.slice;
log = (typeof console != "undefined" && console !== null ? console.log : void 0) || function(o) {
  return alert(o);
};
printResults = function() {
  var r, runners, winner, _i, _len, _results;
  winner = arguments[0], runners = 2 <= arguments.length ? __slice.call(arguments, 1) : [];
  log("Winner:", winner);
  if (runners != null) {
    _results = [];
    for (_i = 0, _len = runners.length; _i < _len; _i++) {
      r = runners[_i];
      _results.push((function(r) {
        return log("Runner up", r);
      }));
    }
    return _results;
  }
};
printResults("James", "Paul", "Martin", "John");

Look familiar to you .NET people? It should because for all intents and purposes this is the same as the param keyword. Implementing this sort of thing in JavaScript is no easy task. Again the generated JavaScript could be simplified if written by a human bean but could you get it right first time without seeing this generated code?

Destructing Assignment

Already part of the JavaScript spec but not present in all implementations (see MDC article) destructing assignments allow the assigning of multiple values in one fell swoop e.g.

# Destructing Assignments
a = 1000
b = 0

[a, b] = [1000, 0]
# a now = 0, b now = 1000 

# Multiple return values
weatherReport = (loc) -> [loc, 20, "Sunny"]

[city, temp, forecast] = weatherReport "Belfast"

# city = "Belfast"
# temp = 20
# forecast = "Sunny"

//-- GENERATED JS 

// Destructing Assignments
var a, b, city, forecast, temp, weatherReport, _ref, _ref2;
a = 1000;
b = 0;

_ref = [1000, 0], a = _ref[0], b = _ref[1];
// a now = 0, b now = 1000 

weatherReport = function(loc) {
  return [loc, 20, "Sunny"];
};
_ref2 = weatherReport("Belfast"), city = _ref2[0], temp = _ref2[1], forecast = _ref2[2];
// city = "Belfast"
// temp = 20
// forecast = "Sunny"

I've never really come across a massive need for something like this but I guess it's another handy tool to have in the scripting world.

Should We All Be Using It?

If you think I am going to say "Yes" immediatley you are wrong. And if you are now asking yourself why I even bothered typing this.... lets just say train rides are boring but not as boring as most of the TV shows my darling wife makes me sit through EVERY FLIPPING NIGHT OF MY LIFE!!!!

The big reason I am not recommending it yet is that I am not 100% convinced of the benefits of using it above plain old JavaScript. It's a bit like marmite. Many node.js developers write in CoffeeScript, and ONLY CoffeeScript but many still simply reject it. I can understand both standpoints so until I can say without faltering that one side has got it right I am not taking it seriously. That said I prefer to form my own opinions anyway so I need to spend a lot more time with it before I can jump to any conclusions.

Oh and, honestly, who actually really wants to spend what limited time they have trying new things, hacking around, experimenting, taking risks, getting frustrated and making plenty of mistakes? Anyone? Anyone at all? Nah I didn't think so. Computers are just a 9-5 job for all of us ;-P

Node.js Lightning Talk

Thu Mar 10 00:00:00 UTC 2011

So I held a lightning talk on node.js the other day. A lightning talk is essentially a short (15 minute) talk about a subject with a bit of discussion at the end of it. There was a slight technical hitch in that I forgot to record the session but I have made up for it by sitting alone in a quiet room early this morning a re-recorded the session – minus the actual discussion bit.

Obviously it's not as good as the original talk but thought it might be worth putting out there anyway as it's better than the slides alone. So here it is – a very high level view of node.js.

Video

Slides

Mobile Redirection for Web Applications in ASP.NET MVC

Thu Feb 24 00:00:00 UTC 2011

Mobile redirection is simple stuff but what happens when you need to deep link into the mobile application?

On a recent project we needed to produce mobile (iPad specifically) equivalent. The desktop app itself was developed using ExtJS (3.3.1) and had three primary entry points,

Logon Screen
Home Screen
Product Screen (New/View/Edit)

Due to the way the project was implemented these became as 3 distinct pages/controllers/actions. Users could receive emails with links to a particular product and they would go directly to that view (with a login redirect if not previously authenticated). The mobile solution, however, written using Sencha Touch (consistent development experience, native-esque UI with little effort) is a single page application. This presents a problem when the user is on a compatible mobile device and they receive a link to a particular placement - how do we push that sort of deep linking into a single page app. Well on the client side frameworks such as Backbone.js, jQuery Mobile and Sencha Touch [anyone got more please?] all offer history support using hash navigation. That's the client side sorted but how do we translate, say, /Product/Show/12345 into /Mobile#placement/12345?

MobileRedirectAttribute

Firstly I created an extension of the AuthorizationAttribute that will act as an interim redirection and request parser between the mobile and desktop solutions. Here's the code (usage follows),

/// <summary>
/// Redirects to the mobile view if on a supported device
/// </summary>
[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method, Inherited = true, AllowMultiple = false)]
public class MobileRedirectAttribute : AuthorizeAttribute
{
    private string _clientFragment;

    /// <summary>
    /// Default Constructor
    /// </summary>
    public MobileRedirectAttribute()
    {
        _clientFragment = string.Empty;
    }

    /// <summary>
    /// Constructor that takes an argument
    /// </summary>
    /// <param name="clientUrl">The url fragment we should append to the url</param>
    public MobileRedirectAttribute(string clientFragment)
    {
        _clientFragment = clientFragment;
    }

    /// <summary>
    /// Tests if this request originates from a supported mobile device 
    /// and redirects as appropriate
    /// </summary>
    /// <param name="ctx">The action execution context</param>
    public override void OnAuthorization(AuthorizationContext ctx)
    {
        if (ctx.HttpContext.Request.Browser.IsMobileDevice)
        {
            // parse the fragment with request parameters
            string fragment = ParseClientFragment(ctx);

            // construct the redirect url
            UrlHelper urlHelper = new UrlHelper(ctx.RequestContext);
            string url = string.Format("{0}#{1}", urlHelper.Action("Index", "Mobile"), fragment);

            // return redirect result to prevent action execution
            ctx.Result = new RedirectResult(url);
        }            
    }

    /// <summary>
    /// Parses the client fragment and replaces :[token] with the request parameter
    /// </summary>
    /// <param name="ctx">The controller context</param>
    /// <returns>The parsed fragment</returns>
    private string ParseClientFragment(ControllerContext ctx)
    {
        string parsedFragment = _clientFragment ?? string.Empty;

        if (!string.IsNullOrEmpty(parsedFragment))
        {
            NameValueCollection @params = ctx.HttpContext.Request.Params;
            MatchCollection matches = Regex.Matches(_clientFragment, ":[a-zA-Z]+");
            RouteData routeData = RouteTable.Routes.GetRouteData(ctx.HttpContext);

            // check each token and replace with param or route values
            foreach (Match match in matches)
            {
                string token = match.Value.TrimStart(':');
                string value = @params[token];

                // if we haven;t got a parameter here we must check the route values
                if (string.IsNullOrEmpty(value) && routeData.Values.ContainsKey(token))
                {
                    value = routeData.Values[token] as string;
                }

                // perform the replace
                parsedFragment = parsedFragment.Replace(match.Value, value);
            }
        }

        return parsedFragment;
    }
}

Usage

So for our 3 entry points into our application we attribute the controller actions with the MobileRedirectAttribute and give it a client fragment.

public class ProductController : Controller
{
    [MobileRedirect("[product/:id")]
    public ActionResult Index(Nullable<long> id)
    {
        // perform action
    }
}

public class HomeController : Controller
{
    [MobileRedirect("home")]
    public ActionResult Index()
    {
        return View();
    }
}

public class AuthenticationController : Controller
{
    [MobileRedirect("home")]
    public ActionResult Login()
    {
    }
}

The client fragment is capable of translating tokens embedded within it ( as :<token_name>) and replacing the token with a matching route value or request parameter. The ProductController Index action is a good example of this. A request to /Product/Index/12345 on a mobile device would translate to /Mobile/#product/12345

How It Works

Pretty simple really.

The attribute checks if the device is a compatible/mobile device.
If it is the retrieves the client fragment and extracts the tokens - :<token_name>
It attempts to match the token names against request parameters first
If there is no parameter it then looks into the route values (eg. :id in the above url isn;t a parameter but rather a route value)
It replaces the token with the real value
It performs a redirect to /Mobile#<client_fragment> which cancels the execution of the action.

Other Points

It's probably not the most robust solution in that more complex scenarios may not work as expected but it's a decent base that can be extended.
I have hardcoded the mobile route as it fitted my needs so I think that should be externalised as well.
The determination of whether a device is a compatible device is facilitated through Browser.IsMobileDevice. This is for demonstration purposes only. In the project we use a different solution but it is a bit more long winded to explain here.
The "redirect to logon" handling is performed within the app itself so that is why the Logon view's fragment is simply "home".

Demo

I've pushed a very quick and dirty demo of this onto GitHub for anyone interested - https://github.com/kouphax/mobileredirect-mvc/ . It uses a really quick UserAgent.Contains("iPad") check for "mobile" detection so use and iPad or set your User Agent to try it out.

Any use to anyone out there? Any problems with it? Let me know.

Entity Framework: Code First - Head First

Fri Feb 11 00:00:00 UTC 2011

I find the data access layer on most projects to be either overly complex or fiddly with lots of XML mapping files that are difficult to debug so anything that could make this layer more developer friendly I'm all for it. Though I also want to point out that I understand why the DAL is often complex - there is a lot to consider and so I want to also understand if these "friendlier" technologies can handle that sort of complexity or if they simply make the happy path easier but making the more complex scenarios more difficult or even impossible (which is obviously a blocker).

So I've been tinkering with this new Entity Framework CTP5 release and the "Code First" features recently. This comes after some time-out from EF due to some really bad experiences with EF1. I was promised that there has been significant changes/improvements since I last dabbled and it really seems there have been. So I wanted to put it to the test and as one of my co-workers wanted an "Ideas" app I thought it would be a fun [may not be anyone's definition of fun but my own] to throw an MVC app together using EF "Code First" to model my domain entities. To make it all even more simple I went ahead and used SQL Server CE 4 for persistence. So what did the solution need to do? The basic requirements were,

Use Windows Authentication for users
Allow users to submit an idea (Title, Description)
Allow users to tag an idea with a variable number of tags
Allow users to vote up or vote down ideas (but not their own)
Allow users to comment on ideas
Allow users to filter ideas by tags
Allow users to sort ideas by newest ideas or by most popular.

Nothing too extreme involved - not unless you turn the whole thing into a computer game style EXTREME speed run competition - man vs. machine - the ULTIMATE [typing] battle.... with bathroom and snack breaks! Just to make it even more INSANE I documented my steps and created a graphical timeline of the session in a PRETTY timeline.

Ammmm don't mean to be rude but your jaw.... we'll it's on the floor. Can you pick it up please? 17:21 to 20:38 minus about an hour and a bit for bathroom, snack and chat breaks - zero to datafied in less than 3 hours! Few points to note,

This experiment focused on the data model, EF CTP5 and the database.
There is a working UI (MVC3) it's just not exactly pretty
I had no EF "Code First" experience before hand
I could be doing a few things incorrectly
It'll probably take me longer to write this post than it did the app.

So lets look at what I produced. The source is available on Github* for your fiddling pleasure.

I am not going to dive into the whole MVC part of it as the source is available but I may touch on some of the interface points such as controllers and binders.

The Domain Models

Lets take a high level look at our domain models.

DomainEntity

The abstract domain entity is used to prevent me having to repeat common auditing and database related stuff across all my entities. It is not mandatory or derived from anything related to Entity Framework - all these classes are simple POCO's. DomainEntity sets up the entities primary key using the Key attribute and also gold 2 audit related properties CreatedBy and CreatedDate.

/// <summary>
/// Base class for domain entities responsible for holding auditing and 
/// persistence related properties
/// </summary>
public abstract class DomainEntity
{
    [Key]
    public int Id { get; set; }

    [Required]
    public string CreatedBy { get; set; }

    [Required]
    public DateTime CreatedDate { get; set; }
}

Idea

Idea is our principle class in our domain. As you can see there are various associations set up between the other classes. 2 1-* mappings between Comment and Vote and a *-* mapping between itself and tag (a tag can exist for any number of ideas and an idea can have many tags). It also holds a number of methods related to business logic - specifically calculating Votes, number of Comments etc.

/// <summary>
/// Main domain object in the idea solution.  Represents an idea
/// </summary>
public class Idea : DomainEntity
{
    [Required]
    [MaxLength(255)]
    public string Title { get; set; }

    [Required]
    public string Description { get; set; }

    [DefaultValue(false)]
    public bool IsRejected { get; set; }

    #region Associations
    public virtual ICollection<Comment> Comments { get; set; }
    public virtual ICollection<Tag> Tags { get; set; }
    public virtual ICollection<Vote> Votes { get; set; }
    #endregion

    #region Business Logic
    public virtual int Score
    {
        get { return Votes == null ? 0 : Votes.Sum(v => v.Value); }
    }

    public virtual int VoteCount
    {
        get { return Votes == null ? 0 : Votes.Count; }
    }

    public virtual int CommentCount
    {
        get { return Comments == null ? 0 : Comments.Count; }
    }
    #endregion
}

Tag

Tag is pretty simple. The only interesting thing about it is the use of NormalisedName - essentially the name field lowercased and whitespace removed. This is used when attempting to fetch potentially existing tags from the database.

/// <summary>
/// Represents a tag in the idea solutuion
/// </summary>
public class Tag : DomainEntity
{
    private string _name;
        
    [Required]
    public string Name 
    {
        get
        {
            return _name;
        }
        set
        {
            _name = value;
            NormalisedName = value.ToLower().Replace(" ", string.Empty);
        }
    }

    [Required]
    public string NormalisedName { get; set; }

    #region Associations
    public virtual ICollection<Idea> Ideas { get; set; }
    #endregion
}

Vote

Rather than just store a calculated value against an idea the Vote object represents a rich representation of a Vote (either up or down, whom by and when). This allows us to provide extra validation when we need it. For example people not allowed to vote on their own idea or vote on an idea in any particular direction more than once. Having this rich association makes these things much easier and we aren't forced to create custom objects to track this sort of thing.

/// <summary>
/// Represents a single vote for an idea
/// </summary>
public class Vote : DomainEntity
{
    [Required]
    [Range(-1,1)]
    public int Value { get; set; }

    #region Associations
    public virtual Idea Idea { get; set; }
    #endregion
}

Comment

Nothing special here.

/// <summary>
/// Represents an ideas comment
/// </summary>
public class Comment : DomainEntity
{
    /// <summary>
    /// Gets or sets the comments content
    /// </summary>
    [Required]
    public string Text { get; set; }

    #region Associations
    /// <summary>
    /// Gets or sets the link to the parent idea
    /// </summary>
    public virtual Idea Idea { get; set; }
    #endregion
}

The Database Context

This is where all the EF magic happens. We use this class to provide an entry point into our database. It's possible to configure entities here in terms of mapping and associations as well as providing a means to seed the database with initial data but I didn't need any of that. No I just defined my sets and added a method for filtering/sorting ideas based on criteria. Simple stuff yet again. It just extends the DbContext class from Entity Framework.

/// <summary>
/// Data Repository for the ideas solution
/// </summary>
public class IdeaRepository : DbContext
{
    #region Db Sets
    public DbSet<Comment> Comments { get; set; }
    public DbSet<Idea> Ideas { get; set; }
    public DbSet<Tag> Tags { get; set;}
    public DbSet<Vote> Votes { get; set; }
    #endregion

    /// <summary>
    /// Main entry point for querying the ideas dat
    /// </summary>
    /// <param name="filters"></param>
    /// <returns></returns>
    public IList<Idea> QueryIdeas(IdeaFilter filters = null)
    {
        IQueryable<Idea> ideas = Ideas;
        IdeaFilter.OrderBy orderBy = IdeaFilter.OrderBy.MostVotes;
        if (filters != null)
        {
            if (!string.IsNullOrWhiteSpace(filters.Tag))
            {
                ideas = ideas.Where(i => i.Tags.Any(t => t.NormalisedName == filters.Tag));
            }

            orderBy = filters.Order.GetValueOrDefault(IdeaFilter.OrderBy.MostVotes);
        }

        switch (orderBy)
        {
            case IdeaFilter.OrderBy.MostVotes:
                ideas = ideas.OrderByDescending(i => i.Votes.Sum(v => v.Value));
                break;
            case IdeaFilter.OrderBy.Newest:
                ideas = ideas.OrderByDescending(i => i.CreatedDate);
                break;
        }

        return ideas.ToList();            
    }
}

Tag Model Binder

This was an interesting thing I discovered. If you are using a *-* relationship and are associating one side with an object that already exists you are required to fetch this object before using it. For example when adding a tag to an idea I need to attempt fetch that tag first of it exists. What I can't do is create a new tag object and assign an existing Id to it - this will be thrown away and saved as a new instance. To fix this problem I feel back onto a Tag Model binder that attempts to fetch or create tags depending on their normalised name. It won't save new tag - simply create them (this is why I use a shared DbContext between the controller and the binder). The binder takes a CSV styled string, breaks it apart, "normalises" the string and tries to fetch tags based on their normalised name. If it finds one it pushes it into the collection otherwise it creates a new tag object and pushes that in instead. Probably a better way to do that and I am open to suggestions. But what I don't want is saving tags that are then going to become orphaned if the other save didn't go through for some reason.

/// <summary>
/// Converts a string of tags (comma seperated) into a list of tags - 
/// creating new ones where necessary and fecthing exisitng ones
/// </summary>
public class TagCollectionModelBinder : DefaultModelBinder
{
    public override object BindModel(ControllerContext controllerContext, ModelBindingContext bindingContext)
    {
        List<Model.Tag> tags = new List<Model.Tag>();            
        HttpContextBase ctx = controllerContext.HttpContext;
        string user = ctx.User.Identity.Name;
        string value = ctx.Request.Form[bindingContext.ModelName];
        Model.IdeaRepository _db = Utils.BaseIdeaController.DataContext;

        if (!string.IsNullOrWhiteSpace(value))
        {
            string[] clientTags = value.Split(',');
            foreach (string clientTag in clientTags)
            {
                string normalised = clientTag.ToLower().Replace(" ", string.Empty);
                Model.Tag tag = _db.Tags.FirstOrDefault(t => t.NormalisedName == normalised);

                if (tag == default(Model.Tag))
                {
                    tag = new Model.Tag()
                    {
                        CreatedBy = user,
                        CreatedDate = DateTime.Now,
                        Name = clientTag.Trim()
                    };
                }

                tags.Add(tag);
            }

            return tags;
        }

        return null;
    }

}

Other Things

Not very much else worth mentioning right now as I more or less used the MVC scaffolding to build the views (with some minor tweaking). Validation on the UI is pushed from the domain object making things a lot more streamlined. The controllers are still very light and could be made lighter by pushing stuff into the IdeaRepository as well but that's for another day.

So there you go. A very quick and dirty intro into the world of Entity Framework. There isn't anything complex going on here and I was worried that EF would mask a lot of stuff that we would need access to but it seems there is plenty of configuration points to hook into. It has come on leaps and bounds since I last dipped my toes into EF and hopefully they keep up the same momentum. There is still a lot of due diligence required before I'd recommend EF over any other data access layer that we are currently using but I am certainly keen to dig deeper and push it to it's limits.

Once again the source for the solution is available on Github*. Phew.....

* Expect bugs.

MvcConf2: Post Conference Rubdown

Wed Feb 09 00:00:00 UTC 2011

So MvcConf has been and gone. Lots of interesting talks though sadly I hit a ton of technical issues with laggy video and eventually total audio/video loss. To my ISP – shame on you! Luckily I did get to see most of the things I wanted (or at least hear). Here's the final outlay of talk I attended (green for attended, yellow for not attended),

I kind of wanted to attend,

Talks that I already had some experience in to see if my thoughts where aligned with others, and,
Talks that I had not experience in but had an interest in.

Problem was that in some of these talks I felt a bit out of my depth which wasn't helped by the fact my video was busy drawing part of slides 4 and 5 when the presenter was already on slide 7 :(. Other times I felt I had already dived deeper than the talk was covering. I guess it happens! Anyway enough complaining lets see what we covered,

Real World Application Development with Mvc3 NHibernate, FluentNHibernate and Castle Windsor

Presenter: Chris Canal

I went into this talk wanting to get a quick glance at NHibernate specifically. I've had no exposure to any of these technologies (bar MVC3 of course) and I think I suffered because of it. It was very Castle Windsor focused and I think I would have been better jumping into a pure NHibernate talk if there was one. That said I really should have a look at Castle Windsor.

BDD in ASP.NET MVC using SpecFlow, WatiN and WatiN Test Helpers

Presenter: Brandon Satrom

I recently attended a talk on using Cucumber to provide Automated UI testing on our projects (held internally in our company) so I wanted to see what SpecFlow brought to the table over running Ruby and Cucumber. I was very impressed with what I seen. Nice explanation of the theory behind BDD. I like the whole “Working Software” vs “Working Software That Matters” comparison.

In terms of SpecFlow itself I like the tighter integration into .NET, the ability to debug your steps is very nice and might help increase acceptance in a team when compared to introducing Ruby. I can see the arguments for using Ruby – Cucumber is a more mature, Ruby is faster to setup and tear down, using the right tool for the right job etc. but I can also see the arguments against. At least both farmeworks use the Gherkin language so writing the scenarios is done in the same way. This is something I have to mull over a bit more.

Quality Driven Web Acceptance Testing

Presenter: Amir Barylko

I had a few audio and video problems during this presentation. It touched on some theory behind Acceptance testing, Green Field vs Brown Field and things like that then dropped into using Cucumber for some acceptance testing. Having just attended the same type of talk a few days previous there wasn't anything new here, and given the bad connection I skipped the rest of the talk.

Keynote

Presenter: Scott Guthrie

The Gu talking about stuff – nuff said. Nice to see him describing how cross pollination of the three platforms (WebMatrix, WebForms, MVC) works (Razor from WM, Routing from MVC etc.). Also nice to hear that we can expect an MVC release on a near annual basis with plenty of preview releases inbetween!

CQRS and Event Sourcing with MVC 3

Presenter: Ashic Mahtab

Wow – CQRS is something I'd heard of previously and it's a pretty big buzz word in the Silverlight world at the minute so I wanted to check it out. My head was thoroughly melted after this one. Quite fast paced and I could have benefited from getting some prior reading in and around the CQRS concept. Will revisit this once the video and slides are available.

One unexpected thing I got out of this talk was MSpec. I always wondered how things like cucumber and spec flow handled lower level unit testing and it seems they aren't really suited to that sort of thing. MSpec, as I have discovered, is essentially a unit level focused BDD framework. Going to check this out very soon.

Deploy ASP.NET MVC with No Effort

Presenter: Troels Thomsen

Honorable mention here as my connection was flat out dying on this room. I couldn;t make out video or audio very well. I did spend a few hours a few days ago looking at appharbor and it looks like, as the name says, “Azure done right”. Love the use of git as a deployment framework, love the minimal interface, love the pricing model, just love the whole thing – so I'm sorry I had to miss the presentation. Another one to check out later.

Entity Framework "Code First": Domain Driven CRUD

Presenter: Chris Zavaleta

Having been playing with EF CTP5 recently I wanted this one to see if there was anything worth expanding on. Turns out there is some nice stuff I didn't see during my tinkering. Especially around the use of Configuration Categories to break apart the OnModelCreating configuration and the Seeding stuff to populate the DB with data. Again my audio was dropping off constantly and my video was struggling so I had to drop out early on this one and throw in the towel.

Summing Up

All in all a decent conference I could have done with reading up on some of the subjects before hand. I will be visiting many of talks again when they come online to get a better picture (pun intended) of the topics. Big thanks to all involved.

jQuery 1.5 Released

Tue Feb 01 00:00:00 UTC 2011

Official Post: http://blog.jquery.com/2011/01/31/jquery-15-released/

Quick post just spread the word that jQuery 1.5 dropped today which is right on time according the the schedule :) . I guess it's a bit pointless because the people that care already know this, but humour me!

New Stuff

There isn't a heap of "new to jQuery" stuff in this release which I guess is a good thing. The MS contributed plugins have stayed just that - plugins. Previous talk of templating becoming absorbed into the jQuery core in 1.5 has failed to bear fruit. Good stuff - not sure why it should have been a core function - there is no one right templating strategy/engine and putting one in the core is sure to annoy as many people as it will please others. So what's new?

Deferred Objects

Deferred Objects are objects that return a contract that will be adhered to regardless of when actions are bound to it. This forms the heart of the Ajax rewrite which I'll touch on in a minute. This is essentially the Promise implementation pattern. Basically when you execute a function that returns a Deferred object you can keep attaching callbacks to that object and they will get executed even if they are bound after the function completes whatever operation it has to do. The jQuery example demonstrates this very well,

// Assign handlers immediately after making the request,
// and remember the jxhr object for this request
var jxhr = $.ajax({ url: "example.php" })
    .success(function() { alert("success"); })
    .error(function() { alert("error"); })
    .complete(function() { alert("complete"); });

// perform other work here ...

// Set another completion function for the request above
jxhr.complete(function(){ alert("second complete"); });

So basically at any point in time we can attach callbacks to the ajax promise, pass it around our system, let other functions attach handlers etc and it will all get resolved/executed when the call has completed. It's nice stuff though it may not have a TON of uses.

jQuery.sub()

Similar in a way to the YUI3 sandboxing technique jQuery.sub() lets us create a clone or subclass of the global jQuery object and hack it to bits for our own needs without affecting the global jQuery object. Again limited uses but when you need it it can be a godsend. Here is a very contrived example,

(function($){
    var sub$ = $.sub();

    sub$.ajax = function(){
        // some sort of stub for ajax testing
    };

    // run tests involving ajax here
    // ... 
})(jQuery);

You see what I am doing? Say I wanted to perform some sort of unit test involving calls to the Ajax method. I could, in isolation, make the Ajax method return whatever I want without worrying about breaking other tests.

Improved/Refactored Stuff

Ajax Rewrite

The ajax module has been rewritten entirely. It will now return a special jQuery based XHR object (rather than a plain XHR object). This object is a deferred object that I spoke about above. This means we can pass the ajax request around and let components or modules attach their own handlers when they need to. We can also bind multiple callbacks via the jQuery chaining pattern (see example above)

Moar Speed

As usual many of the DOM traversal methods have been given performance tweaks - .children(), .prev(), and .next(). The official post has shiny graphs and such so hop on over there for a full run down.

Conclusion

I hate that word but I don't have another word I like better.... Yes so I haven't played with this a great deal recently, just some superficial tinkering but it's a solid release that adds some nice features (with limited uses for most I suspect). I mean they could have kept deferred objects internal - they were implemented for the Ajax stuff - but why not make it public - could be useful to many people, and that is a good attitude.

Also good decisions were made to keep templating out of the core I think (others may disagree but that's life!). Finally this is going to be the jQuery version used in jQuery Mobile when it hits shortly!

.NET Web Technology Stack in 2011 and Beyond....

Mon Jan 24 00:00:00 UTC 2011

Change, or lack thereof is often an indication of how successful your technology stack is. If you are constantly swapping out core technologies then chances are you'll need to rethink your approach. If, on the other your stack remains fairly stable save a few new additions or tweaks then odds are you made the right choices at the start.

Last year, after careful consideration, I published an internal post that detailed my recommendations for the standard Microsoft Web Technology stack. Looking back on this I feel happy enough that my choices have proven themselves. But time goes on and technologies change so perhaps it's time for another look at the stack. Perhaps give it a bit of a refresh after all there has been a number of great developments in the .NET world. Do I need to refresh it? No not really – I could quite easily stick with my previous stack and truth be told the changes are fairly minimal – mostly version bumps. But I also want to look to the future, is there anything worth keeping an eye on (there is BTW), is there anything we should be avoiding (probably) etc.

Recap

Lets take a very quick run through of the recommended stack from last year.

The choice of client side technology really depended on the project requirements. Essentially I suggested most projects use jQuery and some jQueryUI stuff. If the UI required a higher degree of complexity/richness I recommended moving towards Sencha and Silverlight with Silverlight being the preferred option unless there was some project requirement that blocked the use of browser plugins. That's the general gist anyway – I can elaborate if anyone wants to hear more.

Refresh

So I'm recommending a few minor tweaks in an attempt to keep the stack up to date. After all .NET4 and Visual Studio 2010 have dropped since I last made my recommendations. Here goes

Not exactly a major overhaul and it shouldn't be. The stack is simple enough to handle many types of project and has been fairly successful since it was put forward. So I wanted to keep moving in that direction while bumping the versions up to the latest and greatest. So lets attempt to justify some of these changes.

MVC3

Look at the past posts on this blog and you will see I've spent a lot of time with MVC and, more recently MVC3 (Beta – RC – RTM). This blog doesn't cover half the MVC hacking I do on a daily basis. So MVC3 is just released and that might make some people nervous about using it as the primary basis for a web stack. I'm not - it's a quality release both in terms of features and stability. MVC3, for the most part, should be considered more as a refinement of MVC2 with a nice new shiny ViewEngine slapped on top. There really isn't much there that couldn't be done in MVC2 with a bit of custom code. But that's good it kind of hints that they were on the right path to begin with. But the enhancements it does offer make development a much cleaner experience. I already have a heap of blogs on the subject so if you want to know more about my views on MVC3 have a look at these,

Razor View Engine

Razor is Microsoft's internally developed View Engine. It could be seen as a successor to the old clunky WebForms View Engine. I am not going to get into the syntax of Razor and how it is much cleaner than the WebForms syntax, no, if you want to digest all that have a look at one of my earlier posts – The Razor View Engine. In that post I also discuss the differences between Razor and Spark – my previous View Engine recommendation. So what has changed? Why am I recommending Razor over Spark now? Well as the previous post says it best with these lines,

[Spark is] a great engine but has suffered from a lack of tooling.
Spark, especially in comparison to Razor, has very limited tooling support in VS2010 – sure you have SparkSense but it's currently not a patch on Razor support (which is still in itself RC status) and I can't help but feel they'll always be playing catch-up with in house developed/supported stuff.

That view hasn't changed. Razor has a brighter future simply because of the in-house support being offered by Microsoft. Also there isn't an official release of Spark that works against MVC3 (still built against MVC2 dlls) – it is possible to grab a branch off Github though. It's a shame though and I wish all the best for Spark in the future – perhaps our paths will cross again?

The last thing I want to mention about view engine choice – or at least ask – is does it matter? The answer, of course, is not really. Your views should be very light and not have a lot of serverside logic built into them. So sticking with one view engine isn't going to ruin anyone's life.

jQuery/jQueryUI

OOTB integration with MVC is the most obvious reason for sticking with jQuery here. Also the fact that it has proven popular in many of our previous projects means it'll be sticking around for some time. The majority of our companies web projects can be catered for by jQuery (and sometimes including jQueryUI for some richer elements) as a decent base to write custom code. It may not be necessary in all projects of course and shouldn't be abused but that will depend on the project but I am confident enough to say 95% of our web based projects (not including mobile which I will touch on shortly) will feature jQuery and a few plugins to get the job done. All we need to do is bump the version up to the latest and we are flying.

Sencha (version 3)

OK I am still recommending Sencha/ExtJS for the small niche of projects that need a lot of richness but can't make the move to Silverlight. As before I am caveating this (or covering my ass). Sencha is a complex beast it has a steep learning curve and simply hacking away at it in a project is never going to be the best approach. It's best to avoid it but if jQueryUI doesn't cut it and Silverlight can't be deployed make sure the developers have plenty of time to get up to speed, have a capable mentor and code is checked for any bad smells early and often. Done right Sencha can produce brilliant results – but getting it right can be tricky.

Revolution

As always on the Web front things are changing fast. There new techs and new versions always round the corner. Some will be relevant, some just novelty. Obviously to stay ahead of the game we want to be keeping an eye on the technologies and applying some scientific rigour to determine if these should be brought into the recommended web stack. Here is a very quick rundown of my “ones to watch in 2011”.

Knockout.js & Backbone.js

Complex UI require a bit of structure or you'll end up with a lovely pile of spaghetti, the bad kind not the tasty good kind. jQuery and jQueryUI don't really offer anything in this area – sure you can get a collection of plugins to solve this but the best approach is a single unified lightweight framework that does one thing in a simple and consistent manner. Both Knockout and Backbone provide this but in via different routes. Knockout.js offers an MVVM framework similar to Silverlight/WPF data binding while Backbone offers a minimal MVC framework. It's hard to say at this point which is better – both have their pros and cons but that is for another post.

jQuery 1.5

http://blog.jquery.com/2011/01/14/jquery-1-5-beta-1-released/

Looks to be more betterness from the jQuery team (yes i know it's not a real word). So far there has been a re-architecting of the Ajax component so that you can bind callbacks at anytime (thanks the Promise pattern). This pattern is likely to make it into the public API as well so your own can make use of the promise construct. Another new feature appear to be subclassing of the jQuery object. Similar to the YUI3 sandboxing approach subclassing allows us to extend our own instance of the jQuery object without polluting the global instance. These changes combined with a number of bug fixes and performance improvements means jQuery vNext is looking good.

Ext JS 4

http://www.sencha.com

Ext JS 4 promises to be a big improvement over the current versions. Many things have been simplified and refactored which is good because often in version 3+ you get the feeling that things are perhaps OVER-engineered and needlessly complicated. Ext 2/3 also feels like it has massive functionality holes – there are nice widgets but the data package and application architecture always felt half baked. Not with 4 it seems. The data package is shaping up very nicely and they have their own MVC based classes built right into the framework. This could make large applications much easier to manage.

jQuery Mobile & Sencha Touch

Mobile is still an immature area among big companies. Many of them have native apps in various app stores etc. but they still don't really know how best to manage them. Good web based mobile frameworks appeal to them more as it gives them much greater control over the content – after all they own the hosting servers etc. jQuery Mobile and Sencha Touch have address different concerns – jQuery Mobile focuses on offering a consistent cross device experience while Sencha Touch aims to provide a native look and feel on WebKit based browsers (iPhone/iPad/Playbook/Android devices). I expect these frameworks to come into their own this year as they move out of immaturity and more people start experimenting.

Nuget

http://nuget.codeplex.com

A nice little package manager for the .NET world offering public and locally hosted repositories. What makes it different to other is that a package could be anything – not just DLL. I like this and one of the reasons why I like it is that I can use it to distribute my JavaScript frameworks without have to maintain a big list of files needed for each one. I just click jQuery Mobile and it will pull in jQuery and the necessary CSS. I want Backbone and it will pull down Underscore for me as well. Same goes for any .NET libraries. Combine this with the ability to version packages etc. and it will make projects less of a “DLL Hell”.

WCF HTTP

http://wcf.codeplex.com/wikipage?title=WCF%20HTTP

The new WCF HTTP stuff is looking really really nice. One of the big issues with WCF is all the XML configuration and other blah that goes along with it. Newer WCF has given us .svc-less services generated from pure code, better configuration control and now some really nice improvements to the WCF communication channel. You can have custom Media Processors that let you control what is served to the client based on requested content types or actions (JSON, XML, Images – anything you can think of). The cleaning up of UriTemplates also mean that you can produce a very lightweight RESTful service or MVC like architecture very very easily.

Conclusion

Another long ramble that's taken me longer than expected to write up! Obviously there is plenty there to think about and hopefully it will generate some discussion. Hopefully I get a chance to flesh a lot of this out and offer more low level justification on my tech choices. 2011 is shaping up to be a great year in the tech world.

Using CoffeeScript in .NET

Mon Jan 17 00:00:00 UTC 2011

I've been playing with node.js for a while now and some of the things coming out of that community are simply amazing. One of the interesting techs is CoffeeScript which is essentially a JavaScript dialect that compiles into JavaScript. It's goal is to simplify the JavaScript language by removing all those bad parts that can cause a lot of grief and taking all those common tasks (array manipulation, scoping etc and making them very very easy to achieve. I have an old post I have yet to publish externally on the subject but I will get it out there ASAP.

CoffeeScript comes in 2 variations - a node.js specific version and a "standalone" version that can be run in browser (pure JavaScript without any of the node.js assumptions basically). Considering it's parsing one language into another it's not recommended to be running the script client side every time so if you want to use it you want to be doing the parsing server side and possibly using some sort of caching to prevent redoing the parsing on every request. This is the bad(ish) news for .NET folks because there really isn't any stable JavaScript implementations that can run a script as complex a CoffeeScript (this is essentially a compiler of sorts). Well at least not until now.... When i first investigated this I tried using a number of JavaScript engines,

IronJS: Currently only a partial implementation and doesn't appear to run the CoffeeScript compiler fully.
JavaScript.NET: Worked for version 0.9 or so of CoffeeScript but there is a bug that causes version 1.0 to fall over and development appears to have ceased.
Command line JScript: Epic fail. Just wouldn't do what I needed it to.

Then along came yet another JavaScript engine for .NET so I gave it a shot. On the surface Jurassic appeared to offered everything I needed and upon diving a bit deeper I discovered that, yes, it does indeed offer everything I needed - it runs the CoffeeScript compiler. Sweet.

So lets write the compiler wrapper. Pretty basic really,

/// <summary>
/// Processes CoffeeScript files into javascript
/// </summary>
public class CoffeeScriptProcessor
{
    private static readonly string COMPILE_TASK = "CoffeeScript.compile(Source, {bare: true})";

    [ThreadStatic]
    private static ScriptEngine _engine;
    private static object _o = new object();

    private static ScriptEngine Engine
    {
        get
        {
            if (_engine == null)
            {
                _engine = new ScriptEngine();
                _engine.Execute(Resources.CoffeeScriptSource);
            }

            return _engine;
        }
    }

    /// <summary>
    /// Precesses contents as a coffeescript file
    /// </summary>
    /// <param name="contents">The javascript contents</param>
    /// <returns></returns>
    public static string Process(string contents)
    {
        lock (_o)
        {
            try
            {
                Engine.SetGlobalValue("Source", contents);
                return Engine.Evaluate<string>(COMPILE_TASK);
            }
            catch (Exception e)
            {
                return null;
            }
        }
    }
}

Few points worth noting here,

I create a ThreadStatic instance of the Jurassic Engine and feed the CoffeeScript compiler source into it once. The executions/evaluation of that script is quite slow so I only wanted to take the hit once. Jurassic isn't thread safe so I made the reference ThreadStatic.
Just to be extra safe I am locking during the execution of the compilation task.
Any errors just return null. This is a bare bones implementation I have stripped out custom logging and profiling code and an interface that means little in this context.

I have created a VS2010 MVC3 solution that makes use of this processor to generate JavaScript from a simple CoffeeScript example. Have fun.

https://github.com/kouphax/coffeescript-dotnet

What's Next?

This is obviously only the start - if I wanted to use CoffeeScript in a .NET environment I am going to need to consider how it is used. When should I be compiling it? How should I be caching it etc? The usual things. Hopefully I'll touch on these in a later post.

So right now this is kind of a request for comments, this was knocked up quite quickly and likely I have forgotten something or made a massive mistake. Anything you want to offer fire away.

VS2010/MVC JavaScript Intellisense HTML âHackâ

Fri Jan 07 00:00:00 UTC 2011

There is a problem with Visual Studio 2010 JavaScript Intellisense (using MVC in this case) and it is this - If you are referencing JavaScript resources correctly (or Stylesheets for that matter) as you all should be you wont actually get intellisense within an SCRIPT blocks in HTML. So using this sort of script reference,

<script src="@Url.Content("~/Scripts/json2.js")" type="text/javascript"></script>

Will not give you intellisense. Sure you can use JavaScript files and make use of the reference VSDOC tag (in fact maybe you should)

/// <reference path=?~/Scripts/json2.js?>

But this doesn't work (yet) within SCRIPT blocks. So what about those rare times you actually want intellisense within your page? It's actually quite easy though a bit hacky - stick it in a condition branch that never gets reached. So,

@if (false) {
<script src="~/Scripts/json2.js" type="text/javascript"></script>
}
<script src="@Url.Content("~/Scripts/json2.js")" type="text/javascript"></script>

The Intellisense parser doesn't really care about the fact that the code wont ever get hit and you still get your typeahead and documentation etc. but it won't ever get accidentally written to the generated output (as often happened to me). I guess this seems kind of obvious or whatever but perhaps so obvious some people may not have even though about it and hopefully it'll help someone out somewhere at some point sometime, eventually - maybe.

Better JSON Serialisation for ASP.NET MVC

Sat Nov 20 00:00:00 UTC 2010

The Json() method of controllers is a nice little convenience method for serialising server side data into JSON for handy consumption on the client side but it's not without it's issues. Firstly it just cant serialise dates in any nice way due to no hard and fast standard in JSON for dates.

return Json(new
{
    SomeDate = DateTime.Now
});

Simply gets spat back to the front end as a string like so,

"/Date(1290181373164)/"

Not exactly the most useful thing in the world. You are going to need to make use of JSON's reviver callback to parse this accordingly.

Another issue is lack of configurability. C# coding standards recommend the use of PascalCased property names whereas in JavaScript land the standard is camelCase (and many frameworks assume this casing which can cause issues behind the scenes *cough*ExtJS*cough*). You can fix this by adding DataContract and DataMember attributes to your model object but it gets a bit messy and open to error if an attribute is left out accidentally. Or what if you don't have access to the model code?

JSON.NET

Json.NET is my JSON de/serialiser of choice. It's fast and widely configurable plus it fixes the problems I've mentioned above (among many many other things). Not to mention some BSON (Binary JSON) support which could come in handy in the future.

JsonNetResult

So lets fix things. Lets wrap Json.NET up in an ActionResult type and offer a configurable entry point.

/// <summary> 
/// Simple Json Result that implements the Json.NET serialiser offering more versatile serialisation 
/// </summary> 
public class JsonNetResult : ActionResult 
{ 
    public JsonNetResult() 
    { 
    }

    public JsonNetResult (object responseBody) 
    { 
        ResponseBody = responseBody; 
    }

    public JsonNetResult(object responseBody, JsonSerializerSettings settings) 
    { 
        Settings = settings; 
    }

    /// <summary>Gets or sets the serialiser settings</summary> 
    public JsonSerializerSettings Settings { get; set; }

    /// <summary>Gets or sets the encoding of the response</summary> 
    public Encoding ContentEncoding { get; set; }

    /// <summary>Gets or sets the content type for the response</summary> 
    public string ContentType { get; set; }

    /// <summary>Gets or sets the body of the response</summary> 
    public object ResponseBody { get; set; }

    /// <summary>Gets the formatting types depending on whether we are in debug mode</summary> 
    private Formatting Formatting 
    { 
        get 
        { 
            return Debugger.IsAttached ? Formatting.Indented : Formatting.None; 
        } 
    }

    /// <summary> 
    /// Serialises the response and writes it out to the response object 
    /// </summary> 
    /// <param name="context">The execution context</param> 
    public override void ExecuteResult(ControllerContext context) 
    { 
        if (context == null) 
        { 
            throw new ArgumentNullException("context"); 
        }

        HttpResponseBase response = context.HttpContext.Response;

        // set content type 
        if (!string.IsNullOrEmpty(ContentType)) 
        { 
            response.ContentType = ContentType; 
        } 
        else 
        { 
            response.ContentType = "application/json"; 
        }

        // set content encoding 
        if (ContentEncoding != null) 
        { 
            response.ContentEncoding = ContentEncoding; 
        }

        if (ResponseBody != null) 
        { 
            response.Write(JsonConvert.SerializeObject(ResponseBody, Formatting, Settings));             
        } 
    } 
}

As you can see I have exposed the JsonSerializerSettings object allow developers to tune the serialisation all they want. We can also make things even simpler by providing a Controller extension method that we can call without having to create the object directly,

public static class ControllerExtensions 
{ 
    public static JsonNetResult JsonEx(this Controller controller, object responseBody) 
    { 
        return new JsonNetResult(responseBody); 
    }

    public static JsonNetResult JsonEx(this Controller controller, object responseBody, JsonSerializerSettings settings) 
    { 
        return new JsonNetResult(responseBody, settings); 
    } 
}

Simple stuff but still pretty powerful, lets see it in action.

The Date Problem

We can fix the date problem in a number of ways via converters. Converters give us a custom way to convert values of objects and Json.NET provides a number of converters out of the box - IsoDateTimeConverter, JavaScriptDateTimeConverter and an abstract base DateTimeConverter to roll your own. Lets use the JavaScriptDateTimeConverter for this example,

var result = new
{
    SomeDate = DateTime.Now
};

return this.JsonEx(result, new JsonSerializerSettings()
{
    Converters = new List<JsonConverter>
    {
        new JavaScriptDateTimeConverter()
    }
});

The resultant JSON object will generate a JSON object with a JavaScript Date constructor

{ 
    new Date(?) 
}

While not strictly valid JSON native JSON parsers and JSON2 handle it fine.

The Casing Issue

We can use Json.NET's contract resolver to automagically convert PascalCased property names of C# to camelCased property names of JavaScript.

var result = new 
{ 
    SomeProperty = ?my value? 
}; 

return this.JsonEx(result, new JsonSerializerSettings() 
{ 
    ContractResolver = new CamelCasePropertyNamesContractResolver() 
});

Which gives us a response like so,

{ 
    someProperty: ?my value? 
}

Fixed.

Wrap Up

So we have managed to create a much more powerful JSON serialisation technique without having to sacrifice too much of the convenience of the Json() method in the controller. Anyone got any recommendations/enhancements they are willing to share?

The Razor View Engine

Thu Nov 18 00:00:00 UTC 2010

Sorry for the poor syntax highlighting on the source code... not much support for Razor ATM

I've been holding off trying to form an opinion on this until we got the full package so to speak. With MVC 3 RC1 being released last week I guess it's time to share my thoughts on the mildly contentious view engine. First things first though - lets be extremely superficial and compare a a bit of code. Specifically lets look a using a loop to output a list of "really useful" list items. In the older WebForms view engine you'd do something like this.

<ul>
    <% foreach (int index in Enumerable.Range(1, 10))
        { %>        
        <li>Entry <%=index%></li>        
    <%}%>
</ul>

Even for a very trivial example it does look a bit messy - lots of redundant start and end markers etc. Lets look at the equivalent in Razor,

<ul>    
    @foreach (int index in Enumerable.Range(1, 10))    
    {        
        <li>Entry @index</li>        
    }
</ul>

Now excuse me while I go off on a tiny bit of a tangent. There are two things you probably notice about the equivalent Razor syntax,

It's much cleaner because of some very clever parsing
It's still embedding code in a view. THE HORROR!!!!

Yeah I know in this day and age when everyone is trying to eradicate every suggestion of server code in a view Microsoft have went right ahead and actually enabled you to do it in neat/cleaner manner. It works though - doesn't it? I mean why abstract C# when your view is going to get compiled into C#? Why try and mimic HTML/XML when technically speaking your custom markup is pretty much meaningless? What's wrong with being able to see when server side syntax finishes and front end syntax takes over? It makes a lot of sense to me actually.

The Truth Will Out

I've been holding off judgement on Razor for a while until it was more "complete". Truth be told for a long time I could have easily summed it up as

It's the WebForms View Engine with the <% %> replaced with @

In some ways thats true but at an extremely basic level. I misjudged it. I was wrong. I actually kind of like it now.

But what about Spark?

I've been an advocate of the Spark View Engine for sometime in my company and I've used it on a few projects very successfully. It's a great engine but has suffered from a lack of tooling - there are plenty of developers that simply reject or get a negative impression of something because of the lack of tooling for the tech. Spark, especially in comparison to Razor, has very limited tooling support in VS2010 - sure you have SparkSense but it's currently not a patch on Razor support (which is still in itself RC status) and I can't help but feel they'll always be playing catch-up with in house developed/supported stuff. Would I have like to see Spark being adopted by MS rather than them rolling their own view engine? Perhaps, but variety is the spice of life and I'm happy to have Razor around.

So what will I use on future projects? I'm tending towards Razor for now but perhaps that's because it's new and I'm a technology magpie. Also it easier to justify the use of a technology to a customer when they are provided pre-bundled and supported by such a large entity.

</opinion piece>

Back On Track

Now I've had my little digression lets look at some of the features of Razor.

Syntax

As I showed above the syntax is much more terse. The parser is clever enough to detect when you mean server code and when you mean client code. This reduces the need to use a lot of bulky delimiters like <% and %>. Also the parser is able to determine if the @ you just wrote is a server side delimiter or just plain text - most of the time. Alternatively you can fall back on the @@ syntax to use a literal @.

The parser does choke sometimes when you don't wrap text in an appropriate element or tag

@if (true) 
{
    this is a test
}

fails whereas

@if (true)  
{ 
    <span>this is a test</span> 
}

is fine. If you really insist on not using an HTML element you can fall back on one of two things. Using a <text> element that is a Razor tag for marking an area for processing as plain text or a slightly shorter single line equivalent of @:

@if (true) 
{
    <text>this is a test</text>
}

@if (true) 
{
    @:this is a test
}

Layouts and Sections

Razor (obviously) has support for master pages and rending content sections including optional sections. Syntax for it is pretty simple, lets start with our master page _Layout.cshtml

<!DOCTYPE html>
<html>
    <head>
        <title>@View.Title</title>
    </head>
    <body>
        <div id="menu">
            @RenderSection("menu", required: true);
        </div>
        <div id="content">
            @RenderBody()
        </div>
        <div id="footer">
            @RenderSection("footer", required: false);
        </div>
    </body>
</html>

Couple of features worth mentioning,

Use of RenderSection() to render named sections from the view, also able to specify if the section is a required or optional section throught the use of the required argument (i.e. if it needs to or doesn't need to be included in the view during render - if required is true and the section is missing you'll get a runtime error when you attempt to access the page). Another option for optional sections is to use the IsSectionDefined method e.g.
```
<div id="footer">
    @if (IsSectionDefined("footer"))
    {
        RenderSection("footer");
    }
    else
    {
        @:Default Footer Text
     }
</div>
```
This way we can, as demonstrated, provide some sort of default as well.
RenderBody convenience method. This is used to render the body of the view. Unlike WebForms view engine Razor doesn't require you to wrap the main body in a content area - similar to Spark.

So an appropriate view that this master layout can "consume" could look like this

@{    
    Layout = "~/Views/Shared/_Layout.cshtml";
}

<h1>Super Useful Page</h1>
<p>It really is super useful</p> 

@section menu {
<ul>
    <li>Login</li>
    <li>Register</li>
</ul>
}

Notice I have left out the footer but that's not an issue as it has been marked as optional. Also notice that the body is at the root level of the view and sections are wrapped in a section marker. This all adds a more minimal look and feel to the whole view.

_ViewStart.cshtml

Another leaf from the Spark world (and probably many other - sorry to everyone else that supports/implements this sort of feature) is the ability to provide a global place to put code that applies to every view. Using the _ViewStart.cshtml file we could easily just put the Layout declaration from the view code above into a single global location. We can also add helpers and functions there too which I'll talk about now.

Helpers

Helpers enable use to provide context aware reusable HTML "templates" in our views. Imagine a view that listed Top Selling, New and Top Rated products in 3 lists. The most basic way to implement this would be 3 loops like so

<h1>Top Selling Product List</h1>
<ul>    
    @foreach (string product in View.TopSellers)    
    {        
        <li>@product</li>    
    }
</ul> 
<h1>New Product List</h1>
<ul>    
    @foreach (string product in View.NewProducts)    
    {        
        <li>@product</li>    
    }
</ul> 
<h1>Top Rated Product List</h1>
<ul>    
    @foreach (string product in View.TopRated)    
    {        
        <li>@product</li>    
    }
</ul>

But that's not exactly DRY is it? Helpers can help us out here (hence the name I guess :-P). Lets create a helper that does the repeatable stuff for us,

@helper ProductLister(List<string> products){    
    <ul>        
        @foreach (string product in products)        
        {            
            <li>@product</li>        
        }    
    </ul>
}

And the rest of the view...

<h1>Top Selling Product List</h1>
@ProductLister(View.TopSellers) 
<h1>New Product List</h1>
@ProductLister(View.NewProducts) 
<h1>Top Rated Product List</h1>
@ProductLister(View.TopRated)

Much simpler, cleaner, whateverer. Again with the Spark similarities - these are similar to macros.

Functions

Razor also makes it easier to embed arbitrary methods on your page which I guess can be useful for formatting and things like that. This give us the power to extend the Razor view engine and adapt it to suit our needs - powerful yes but also open to abuse as you might guess. Here's a simple function in action,

@functions {
    string Encrypt(string value){        
        char[] asArray = value.ToCharArray();        
        Array.Reverse(asArray);        
        return new string(asArray);    
    }
} 
<ul>    
    <li>Username: @View.Username</li>    
    <li>Password: @Encrypt(View.Password)</li>
</ul>

The function applies a highly technical string encryption technique to a string value returning the result.

Inline Templates

One more feature before I head off to watch telly or sleep or something. Razor allows you to pass little html nuggets or templates as arguments to helpers and functions. There is a details blog post about this over at Vibrant Code and my example is heavily inspired/plagiarised from that. Lets see what we can do,

@functions {     
    IHtmlString Times(int times, Func<int, object> template) {        
        StringBuilder b = new StringBuilder();        
        for(int i = 0; i < times; i++) {              
            b.Append(template(i));          
        }         
        return new HtmlString(b.ToString());    
    } 
}   
<ul>    
    @Times(10, @<li>List Item @item</li>)
</ul>

Look at the second to last line see that little block of html preceded by the Razor marker (@)? See how in our Times function the HTML block it is getting translated transparently as a Func<int, object> the we can call and manipulate in the helper function anyway we want? Can you see how powerful that could be in the right place? Me too. Me too.

Done For Now

There is more to Razor than all this stuff including the ability to use it outside of the view context so it could simply be a templating language if needs be. There is even more than that too but thats for another time. Imagine though.... I started out this blog post more or less telling you I used to think Razor was simply the WebForms view engine but with @'s instead of <% %>'s and hopefully you can see why I was wrong and why I was happy to admit it. My only gripe is that the Razor documentation could be a bit more available. There isn't really a great deal out there ATM. However this has been promised for the official release so I am looking forward to that.

Oh Misfits is about to start......

Underscore.js: The JavaScript Utility Library

Fri Nov 12 00:00:00 UTC 2010

I've been meaning to give Underscore.js some love for some time but never got around to it. The other day I though about writing about Backbone.js (another excellent library from the same people) but didn't feel it was possible to do so without first talking about Underscore as it underpins most of the Backbone.js functionality. So without further ado...

Introduction

Underscore is a utility-belt library for JavaScript that provides a lot of the functional programming support that you would expect in Prototype.js (or Ruby), but without extending any of the built-in JavaScript objects. It's the tie to go along with jQuery's tux.

- http://documentcloud.github.com/underscore/

That's how the creators describe underscore and it's pretty spot on. Underscore offers numerous low level functions that can be used to perform on arrays and collections (as well as few for object and functions). Some of these functions are available in some browsers but not all and because of this Underscore provides a nice layer of abstraction to avoid having to constantly feature detect. Incidentally where a browser does implement a specific function Underscore will always fallback on the native implementation ensuring that performance is as optimal as possible. This is the sort of stuff that you would probably miss if you moved from Prototype to jQuery. Another point to note is that Underscore doesn't extend native objects with it's own implementations which most people will consider a good thing. Instead, like jQuery, Underscore exposes a single global object aptly named _. Now I am not 100% sold on this, I am a bit strange when it comes to neat code and I think this variable name makes code look a bit "noisy". Not a big deal but I like to pick holes in things :-).

Coding Styles

There are two ways to write code using Underscore. Imagine the scenario that given an array of numbers You can go the functional route and call methods directly on the underscore object,

var filtered = _.select([1,4,2,5,78,23], function(v){
    return v > 10;
});

var sum = _.reduce(filtered, function(memo, v){ 
    return memo + v; 
}, 0);

console.log(sum)
// => 101

Or go the OO route for those familiar with the jQuery style syntax,

var sum = _([1,4,2,5,78,23]).chain()
    .select(function(v){ return v > 10; })
    .reduce(function(memo, v){ return memo + v; }, 0))
    .value();
    
console.log(sum)
// => 101

These 2 examples acheive the same thing - filter a list to those values greater than 10 and derive a sum of these values; but the look and feel are actually quite different. Which one you chose will be down to personal preference (I tend towards the former approach but the jury is still out).

Collections/Arrays

Underscore has a ton of collection and array methods - a collection being either an array or an object (as in an object is essentially a property bag/associative array).

The usual suspects for array manipulation/traversal are all here. Coming from a C# background some of the names will be familiar and some are different but they achieve the same thing - here is just a tiny subset,

each()

Loops over a collection executing a callback on each element. break; support provided by _.breakLoop()

_([2,4,6,8,11,13,17]).each(function(num) {
  if(isPrime(num)){
    console.log("First prime: ", num);
    _.breakLoop();
  }
});
=> First Prime: 11

map()

Creates a new array from a collection where each element can be transformed by a callback

var result = _([2,4,6,8,11,13,17]).map(function(v) {
    return isPrime(v) ? true : false;
});

console.log(result)
//=> [false,false,false,false,true,true,true]

first()/last()

Select the first and last elements from an array

var arr = [2,4,6,8,10];

console.log(_.first(arr));
console.log(_.last(arr));
//=> 2
//=> 4

Functions

Underscore provides a number of methods that can be applied to functions. These include the ability to bind methods to a specific object/context (bind), memoize functions (caching or results to increase calculation speeds), delay and defer execution of methods and perform some AOP style actions on functions (wrap and compose).

Objects

Underscore provides a number of common object related methods such as the ubiquitous extend and clone functions. Other useful functions include the use of a tap function. This function lets you "tap" into an execution chain and inspect/manipulate the object in its current state. The underscore example demonstrates this perfectly.

_([1,2,3,200]).chain()
  .select(function(num) { return num % 2 == 0; })
  .tap(console.log)
  .map(function(num) { return num * num })
  .value();

//=> [2, 200]
//=> [4, 40000]

As you can see the tap function is used to push the object (in this case an array) through. There is also a fairly robust set of type detection functions.

isEqual, isEmpty, isElement,isArray, isArguments, isFunction, isString, isNumber, isBoolean, isDate,isRegExp isNaN, isNull, isUndefined

Other Things

There are a few other specific methods in Underscore that provide utility functions including,

template - super simple templating function
mixin - extend underscores functionlaity with your own custom methods
times - execute a callback a specified number of times

Conclusion

This is a very high level overview of Underscore (so I can now go and get all excited abuot Backbone) but hopefully you can see it's worth. Have a look at the comprehensive docs that many of my example where derived from and see if there is anything useful for you there.

Remote Validation in ASP.NET MVC 3 RC1

Wed Nov 10 00:00:00 UTC 2010

UPDATE: Added some words around the extra configuration options available

Remote validation has finally landed in RC1 of ASP.NET MVC 3. It's a weird area as more often than not people tend to over complicate something that is really pretty simple. Thankfully the MVC implementation is fairly straightforward by simply providing wiring allowing the jQuery Validation plugin to work it's magic. Basically there is a new Remote attribute that can be used like so.

public class Credentials
{    
    [Remote("Username", "Validation")]
    public string Username { get; set; }
    
    public string Password { get; set; }
}

As you can see we have attributed the Username field with a Remote attribute. The 2 parameters tell us what Action and Controller we should call to perform the validation. This does make me feel slightly uneasy as it kind of feels like you are coupling the controller to the model which doesn't sit right by me. currently sitting on the fence I'll see how it works in real life. Anyway I implemented it like so,

public class ValidationController : Controller
{
    public ActionResult Username(string UserName)
    {
        return Json(Repository.UserExists(Username), JsonRequestBehavior.AllowGet);
    }
}

And thats you - provided you have the necessary client side libraries included of course (jQuery, jQuery Validate etc). and have Client Side Validation turned on (now by default in MVC3).

Configuration

The Remote attribute offers a few nice little configuration options to make things easier. The typical ones are there such as ErrorMessage, ErrorResource etc. but there are a few specific ones as well.

Fields

There may be a case where ding the name and the value of a single form field isn't enough to perform validation. Perhaps validation is affected by some other field/value in the form. The Remote attribute accepts a comma separated list of other fields that need to be sent up with the request using the Fields parameter

This basic example will send up the value of the EmailService input field along with the value of Username. Clean and simple.

[Remote("Username", "Validation", Fields = "EmailService")]

HttpMethod

HttpMethod simply allows us to change how the ajax request is sent e.g. via POST or GET or anything else that makes sense. So to send a remote request via POST

[Remote("Username", "Validation", HttpMethod = "POST")]

A Minor Difference

You might notice if you read the release notes for RC1 that my implementation of the controller is slightly different. The reason being that the example in the release notes is broken :-). The example looks like this

public class UsersController {
    public bool UserNameAvailable(string username) {
        return !MyRepository.UserNameExists(username);
    }
}

However the Validate plugin expects a JSON response which is fine on the surface but returning a boolean response to the client side results in a response body of False (notice the captial F) which in turn causes a parse error when the plugin performs JSON.parse. My suggested solution is actually more inline with how most people would typically write an Ajax capable controller action anyway (though I am not happy with the JsonRequestBehaviour usage) but there are other ways but they aren't pretty....

public class ValidationController : Controller
{        
    public string Username(string username)
    {
        return (!Repository.UserExists(Username)).ToString().ToLower();
    }
}

See? Ugly and plain WRONG (but it will work).

Nice to see this feature finally landing as it can be useful in certain situations.

All Change! ValidateInputAttribute and SkipRequestValidation

Tue Nov 09 00:00:00 UTC 2010

I recently blogged about the new Exclude feature of ValidateInputAttribute in ASP.NET MVC 3 (Beta 2). Well as with most early adopter types I've been shafted! Well not really but things have changed slightly and it makes my last post slightly redundant.

Let me clarify a bit. The Exclude property no longer exists. It has instead been replaced with a new attribute SkipRequestValidation. This is a per-property attribute that lets you specify what properties should be excluded. This has the effect of pushing the validation flag down to the model rather than on the controller. This make a lot more sense especially in a controller/solution that has many actions accepting the model. So what changes? Lets take a look at our old code.

First things first the controller no longer need the ValidateInput attribute and goes back to the old "thinner" version of itself,

public class HomeController : Controller
{
    public ActionResult Post()
    {
        return View();
    }

    [HttpPost]
    public ActionResult Post(ForumPost post)
    {
        return View();
    }
}

We then need to mark the Bodyt property of our ForumPost model to skip validation,

public class ForumPost
{
    public string Subject { get; set; }
    
    [SkipRequestValidation]
    public string Body { get; set; }
}

Thats it. To be honest this feels like a much neater solution. It feels more a solution rather than a tacked on property that was added to cater for a specific case.

ASP.NET MVC 3 RC1 is out now - read more here, and get it here.

Mobilize! Adapting the ASP.NET MVC Project Template for jQuery Mobile

Mon Oct 25 23:00:00 UTC 2010

Image courtesy of the awesome Threadless

Having tinkered with jQuery Mobile and MVC a bit recently but I wanted to do a bit more. The first app I wrote was read only which is actually pretty easy to accomplish regardless of technology/framework. So I wanted to try a little something different, something a kin to a real application. However time and coders block left me with little inspiration for creating the worlds next amazing mobile app. Sitting there with a new ASP.NET MVC project open in Visual Studio I couldn't think of what to write. Then it struck me... don't write anything. The MVC starter skeleton project was sitting right there why not just mobilise it? So I did. Without modifying any server side code and only tweaking the views I found I was able to create a fully jquery-mobilified version of the template. Here's what I did.

Clean House

We need to strip some of the dead weight out of our project including some of the mark up and resources,

Resources

First things first I dropped the current bundle of scripts (MS Ajax, jQuery 1.4.1, Validate etc.) and the Site.css. Next I added the jQuery Mobile assets,

jQuery 1.4.3 (jquery-1.4.3.js)
jQuery Mobile 1.0a1 (jquery.mobile-1.0a1.js)
jQuery Mobile CSS (jquery.mobile-1.0a1.css)
jQuery Mobile images

In the following project structure

Site.Master

Next I cut down the Site.Master to it's bare minimum and add the HTML5 DocType and the references to jQuery Mobile ,

<%@ Master Language="C#" Inherits="System.Web.Mvc.ViewMasterPage" %>
<!DOCTYPE html>
<html>
    <head id="Head1" runat="server">
        <title><asp:ContentPlaceHolder ID="TitleContent" runat="server" /></title>
        <link href="../../Content/jquery.mobile-1.0a1.css" rel="stylesheet" type="text/css" />
        <script src="../../Scripts/jquery-1.4.3.min.js" type="text/javascript"></script>
        <script src="../../Scripts/jquery.mobile-1.0a1.min.js" type="text/javascript"></script>
    </head>
    <body>
        <asp:ContentPlaceHolder ID="MainContent" runat="server" />
    </body>
</html>

I could have included the basic markup for a full page but I wanted to keep it as flexible for now. I also removed the LogOnUserControl for now as I want to just push it into the main page.

Marking Up

The next step was to tweak the markup of the pages so they comply with jQuery Mobile and can be mobilised correctly. Most pages are marked up in the same way so rather than list them all I'll just list the highlights.

Home.aspx

<%@ Page Language="C#" MasterPageFile="~/Views/Shared/Site.Master" Inherits="System.Web.Mvc.ViewPage" %>

<asp:Content ID="Content1" ContentPlaceHolderID="TitleContent" runat="server">
    Home Page
</asp:Content>

<asp:Content ID="Content2" ContentPlaceHolderID="MainContent" runat="server">
    <div data-role="page">
	    <div data-role="header">
		    <h1>Home Page</h1>
            <% if (!Request.IsAuthenticated) { %>
                <a href="<%=Url.Action("LogOn", "Account")%>" data-icon="forward" class="ui-btn-right">Logon</a>
            <% } else { %>
                <a href="<%=Url.Action("LogOff", "Account")%>" data-icon="back" class="ui-btn-right">Log Off</a>
            <% } %>
	    </div>
	    <div data-role="content">	
		    <h2><%: View.Message %></h2>	
	    </div>
    </div>
</asp:Content>

I have a top toolbar button for Login/Log Off (depending on context) and the View.Message from the old page is included as the content of the page.

Logon.aspx

<%@ Page Language="C#" MasterPageFile="~/Views/Shared/Site.Master" Inherits="System.Web.Mvc.ViewPage<Mvc.Mobile.Template.Models.LogOnModel>" %>

<asp:Content ID="loginTitle" ContentPlaceHolderID="TitleContent" runat="server">
    Log On
</asp:Content>

<asp:Content ID="loginContent" ContentPlaceHolderID="MainContent" runat="server">
    <div data-role="page">
	    <div data-role="header">
		    <h1>Log On</h1>
            <a href="<%=Url.Action("Register", "Account")%>" data-icon="gear" class="ui-btn-right"  data-rel="dialog" data-transition="pop">Register</a>
	    </div>
	    <div data-role="content">	            
            <p>Please enter your username and password</p>
            <% using (Html.BeginForm()) { %>
                <div data-role="fieldcontain">
                    <%: Html.LabelFor(m => m.UserName) %>
                    <%: Html.TextBoxFor(m => m.UserName) %>
                    <div>
                        <%: Html.ValidationMessageFor(m => m.UserName) %>
                    </div>
                </div>
                <div data-role="fieldcontain">
                    <%: Html.LabelFor(m => m.Password) %>
                    <%: Html.PasswordFor(m => m.Password) %>
                    <div>
                        <%: Html.ValidationMessageFor(m => m.Password) %>   
                    </div>
                </div>
                <div data-role="fieldcontain">
                    <%: Html.CheckBoxFor(m => m.RememberMe) %>
                    <%: Html.LabelFor(m => m.RememberMe) %>
                </div>
                <input type="submit" value="Log On" data-theme="b"/>
            <% } %>
	    </div>
    </div>
</asp:Content>

Gives us the login form and a link to the Register screen. I marked the Register screen as a Dialog just to be a wee bit different. The Register screen itself is marked up as a normal page (the framework handles the displaying of a dialog automagically). I also added a different theme to the login button and a "pop" transition to the Register dialog through the data- attributes.

One other thing I had to do was add a "fieldcontain" wrapper to the field blocks as there was some layout issues. This is documented in the jQuery Mobile docs.

Register.aspx

<%@ Page Language="C#" MasterPageFile="~/Views/Shared/Site.Master" Inherits="System.Web.Mvc.ViewPage<Mvc.Mobile.Template.Models.RegisterModel>" %>

<asp:Content ID="registerTitle" ContentPlaceHolderID="TitleContent" runat="server">
    Register
</asp:Content>

<asp:Content ID="registerContent" ContentPlaceHolderID="MainContent" runat="server">
    <div data-role="page">
	    <div data-role="header">
		    <h1>Register</h1>
            <a href="<%=Url.Action("Index", "Home")%>" data-icon="grid" class="ui-btn-right">Home</a>
	    </div>
	    <div data-role="content">	
            <p>Passwords are required to be a minimum of <%: View.PasswordLength %> characters in length.</p>		    
            <% using (Html.BeginForm()) { %>
                <div data-role="fieldcontain">
                    <%: Html.LabelFor(m => m.UserName) %>
                    <%: Html.TextBoxFor(m => m.UserName) %>
                    <div>
                        <%: Html.ValidationMessageFor(m => m.UserName) %>
                    </div>
                </div>
                
                <div data-role="fieldcontain">
                    <%: Html.LabelFor(m => m.Email) %>
                    <%: Html.TextBoxFor(m => m.Email) %>
                    <div>
                        <%: Html.ValidationMessageFor(m => m.Email) %>
                    </div>
                </div>
                
                <div data-role="fieldcontain">
                    <%: Html.LabelFor(m => m.Password) %>
                    <%: Html.PasswordFor(m => m.Password) %>
                    <div>
                        <%: Html.ValidationMessageFor(m => m.Password) %>
                    </div>
                </div>
                
                <div data-role="fieldcontain">
                    <%: Html.LabelFor(m => m.ConfirmPassword) %>
                    <%: Html.PasswordFor(m => m.ConfirmPassword) %>
                    <div>
                        <%: Html.ValidationMessageFor(m => m.ConfirmPassword) %>
                    </div>
                </div>

                <input type="submit" value="Register" data-theme="b" />
            <%} %>
	    </div>
    </div>
</asp:Content>

The most complex screen in the app. Nothing special added only some different markup.

Running It

That was it. After adapting the rest of the pages (via manipulating markup) I was left with a mobilized version of the MVC project template. Again I didn't have to make any code changes - validation still worked, forms still worked, login, register, change password - all worked as expected (screen shots from Chrome which has a few issues with rounded corners on fields in jQuery Mobile).

ValidateInputAttribute Changes in MVC3 (Beta 2)

Sat Oct 23 23:00:00 UTC 2010

This post is now redundant with the release of ASP.NET MVC 3 RC 1. SkipRequestValidation is the new ValidateInputAttribute(Exclude=""). Read More.

The ValidateInputAttribute has received a nice little tweak in MVC 3 offering more fine grained control over parameters of a request.

In MVC 2 using ValidateInputAttribute was limited to the request level, that is all parameters in the request where either validated or not. Lets demonstrate this with a simple example - a simple forum posting page,

<%@ Page Language="C#" MasterPageFile="~/Views/Shared/Site.Master" Inherits="System.Web.Mvc.ViewPage<Common.Model.ForumPost>" %>

<asp:Content ContentPlaceHolderID="TitleContent" runat="server">
	Login
</asp:Content>

<asp:Content ContentPlaceHolderID="MainContent" runat="server">
    <h2>Post Question</h2>
    <% using (Html.BeginForm()) { %>
        <p>
            <%=Html.LabelFor(c => c.Subject)%>
            <%=Html.TextBoxFor(c => c.Subject)%>
        </p>
        <p>
            <%=Html.LabelFor(c => c.Body)%>
            <%=Html.TextAreaFor(c => c.Body)%>
        </p>
        <input type="submit" value="Post" />
    <%} %>
</asp:Content>

We don;t care what the controller actions actually do but lets describe them here anyway

public class HomeController : Controller
{
    public ActionResult Post()
    {
        return View();
    }

    [HttpPost]
    public ActionResult Post(ForumPost post)
    {
        return View();
    }
}

Now attempting to post any sort of markup back to the action will result in an exception being thrown,

[[posterous-content:DasrjrabrrcEcDeiHdeG]]

If we wanted to allow markup to go through we can add the [ValidateInput(false)] attribute to the action. The only problem with that is if we only wanted to allow markup in the Body and not the Subject we would have to write our own tests in the controller to prevent this. Not the most ideal or clean solution.

MVC 3 solves this quite simply by extending the ValidateInputAttribute and allowing use to specify exclusions. This means we can have validation turned on but specifically state the we don't want to validate a specific request parameter(s) (e.g. Body).

[ValidateInput(true, Exclude = "Body")]

A very minor tweak that makes a big leap to being able to produce cleaner more readable code.

Unobtrusive JavaScript in MVC3

Fri Oct 22 23:00:00 UTC 2010

One of the "it's not new but it's cool" features that appeared in the ASP.NET MVC3 Beta was Unobtrusive JavaScript (well actually Unobtrusive Ajax and an unobtrusive validation adapter for the jQuery Validation plugin). Before we dive into how this differs from MVC2 lets talk about JavaScript in MVC.

First things first - jQuery is now the defacto standard for any ASP.NET MVC solution. In the MVC2 Project Template you got both the Microsoft Ajax Library and jQuery. This hasn't changed in MVC3 but the Microsoft stuff is only there for any potential backward compatibility issues a solution may have - feel free to just delete these files and embrace jQuery. All the new client side stuff is all facilitated through jQuery which means you wont have to have some other framework on your page just because generated code mandates it.

What Is It?

Unobtrusive JavaScript (in the MVC3 sense) is a strategy that ensures that no JavaScript is embedded within the markup (unless you do it yourself). 100% no generated code muddying your markup. No code islands, no inline event handlers, better handling of failure cases and no dependence on any specific framework. To me, being a web focused developer, this is HUGE and it should be to you too - it is after all considered best practise.

Turn It On

There are two ways to turn Unobtrusive Ajax/Validation on,

1. Web.Config

Within the <appSettings> config node in Web.Config you can specify whether unobtrusive JavaScript is on or off

  <appSettings>
    <add key="enableSimpleMembership" value="false" />
    <add key="ClientValidationEnabled" value="true"/> 
    <add key="UnobtrusiveJavaScriptEnabled" value="true"/> 
  </appSettings>

2 On a Per-Page Basis

Just like EnableClientValidation it is possible to activate unobtrusive JavaScript at a page level.

<% Html.EnableUnobtrusiveJavaScript(); %>

What Does It Do

Best way to show what it does is by comparison to, um, Obtrusive JavaScript. So lets create a scenario in both MVC2 and MVC3 and seeing how they compare. The scenario will be a simple ajaxified login form (Username and Password) with some client validation and no server side magic. Most of this, bar the client side scripts, is the same across both solutions.

The Model

public class Credentials
{
    [Required]
    public string Username { get; set; }
    [Required]
    public string Password { get; set; }
}

Two properties with basic [Required] validation.

The Controller

public class HomeController : Controller
{
    public ActionResult Login()
    {
        return View();
    }

    [HttpPost]
    public ActionResult Login(Credentials credentials)
    {
        return new EmptyResult();
    }
}

Again there isn't much going on here, this all about the client side!

The View

Common Parts

<%@ Page Language="C#" MasterPageFile="~/Views/Shared/Site.Master" Inherits="System.Web.Mvc.ViewPage<Common.Model.Credentials>" %>

<asp:Content ContentPlaceHolderID="TitleContent" runat="server">
	Login
</asp:Content>

<asp:Content ContentPlaceHolderID="MainContent" runat="server">
    <h2>Login</h2>
    <% using (Ajax.BeginForm(new AjaxOptions { OnSuccess = "onSuccess" })) { %>
        <%=Html.ValidationSummary(true)%>
        <p>
            <%=Html.LabelFor(c => c.Username)%>
            <%=Html.TextBoxFor(c => c.Username)%>
            <%=Html.ValidationMessageFor(c => c.Username)%>
        </p>
        <p>
            <%=Html.LabelFor(c => c.Password)%>
            <%=Html.PasswordFor(c => c.Password)%>
            <%=Html.ValidationMessageFor(c => c.Password)%>
        </p>
        <input type="submit" value="Login" />
    <%} %>

    <!-- JAVASCRIPT LIBRARIES GO HERE! -->

    <script type="text/javascript">
        function onSuccess() {
            // logged in, carry on
        }
    </script>
</asp:Content>

A simple view that is the same across both projects the only thing that is going to differ is the actual libraries/scripts that do all the wiring up.

MVC2

    <script src="<%=Url.Content("~/Scripts/MicrosoftAjax.js") %>" type="text/javascript"></script>
    <script src="<%=Url.Content("~/Scripts/MicrosoftMvcAjax.js") %>" type="text/javascript"></script>
    <script src="<%=Url.Content("~/Scripts/MicrosoftMvcValidation.js")%>" type="text/javascript"></script>

Using the out of the box Microsoft Ajax Library (which is now deprecated)

MVC3

    <script src="<%=Url.Content("~/Scripts/jquery-1.4.1.min.js")%>" type="text/javascript"></script>
    <script src="<%=Url.Content("~/Scripts/jquery.validate.min.js")%>" type="text/javascript"></script>
    <script src="<%=Url.Content("~/Scripts/jquery.unobtrusive-ajax.min.js")%>" type="text/javascript"></script>
    <script src="<%=Url.Content("~/Scripts/jquery.validate.unobtrusive.min.js")%>" type="text/javascript"></script>

Using the out of the box jQuery plugins.

Done. That's all we need for our solutions to function. The form will be validated on the client side and submission will be performed through ajax. Both will behave exactly the same but the markup and code they generate will be quite different. I've tidied both up in terms of formatting just so it's slightly easier to read but I've tried to keep the layout style consistent across both.

MVC2 Output

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
    <title>Login </title>
    <link href="/Mvc2/Content/Site.css" rel="stylesheet" type="text/css" />
</head>
<body>
    <a href="/Mvc2/">Home</a>
    <h2>
        Login</h2>
    <form action="/Mvc2/Home/Login" 
          id="form0" 
          method="post" 
          onclick="Sys.Mvc.AsyncForm.handleClick(this, new Sys.UI.DomEvent(event));"
          onsubmit="Sys.Mvc.AsyncForm.handleSubmit(this, new Sys.UI.DomEvent(event), { insertionMode: Sys.Mvc.InsertionMode.replace, onSuccess: Function.createDelegate(this, onSuccess) });">
    <div class="validation-summary-valid" id="validationSummary">
        <ul>
            <li style="display: none"></li>
        </ul>
    </div>
    <p>
        <label for="Username">Username</label>
        <input id="Username" name="Username" type="text" value="" />
        <span class="field-validation-valid" id="Username_validationMessage"></span>
    </p>
    <p>
        <label for="Password">Password</label>
        <input id="Password" name="Password" type="password" />
        <span class="field-validation-valid" id="Password_validationMessage"></span>
    </p>
    <input type="submit" value="Login" />
    </form>
    <script type="text/javascript"> 
    //<![CDATA[
        if (!window.mvcClientValidationMetadata) { window.mvcClientValidationMetadata = []; }
        window.mvcClientValidationMetadata.push({
            "Fields": [{
                "FieldName": "Username",
                "ReplaceValidationMessageContents": true,
                "ValidationMessageId": "Username_validationMessage",
                "ValidationRules": [{
                    "ErrorMessage": "The Username field is required.",
                    "ValidationParameters": {},
                    "ValidationType": "required"
                }]
            }, {
                "FieldName": "Password",
                "ReplaceValidationMessageContents": true,
                "ValidationMessageId": "Password_validationMessage",
                "ValidationRules": [{
                    "ErrorMessage": "The Password field is required.",
                    "ValidationParameters": {},
                    "ValidationType": "required"
                }]
            }],
            "FormId": "form0",
            "ReplaceValidationSummary": false,
            "ValidationSummaryId": "validationSummary" 
        });
    //]]>
    </script>
    <script src="/Mvc2/Scripts/MicrosoftAjax.js" type="text/javascript"></script>
    <script src="/Mvc2/Scripts/MicrosoftMvcAjax.js" type="text/javascript"></script>
    <script src="/Mvc2/Scripts/MicrosoftMvcValidation.js" type="text/javascript"></script>
    <script type="text/javascript">
        function onSuccess() {
            // logged in, carry on
        }
    </script>
</body>
</html>

MVC 3 Output

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
    <title>Login </title>
    <link href="/Mvc3/Content/Site.css" rel="stylesheet" type="text/css" />
</head>
<body>
    <a href="/Mvc3/">Home</a>
    <h2>
        Login</h2>
    <form action="/Mvc3/Home/Login" 
          data-ajax="true" 
          data-ajax-success="onSuccess"
          id="form0" method="post">
    <p>
        <label for="Username">Username</label>
        <input data-val="true" data-val-required="The Username field is required." id="Username" name="Username" type="text" value="" />
        <span class="field-validation-valid" data-valmsg-for="Username" data-valmsg-replace="true">
        </span>
    </p>
    <p>
        <label for="Password">Password</label>
        <input data-val="true" data-val-required="The Password field is required." id="Password" name="Password" type="password" />
        <span class="field-validation-valid" data-valmsg-for="Password" data-valmsg-replace="true">
        </span>
    </p>
    <input type="submit" value="Login" />
    </form>
    <script src="/Mvc3/Scripts/jquery-1.4.1.min.js" type="text/javascript"></script>
    <script src="/Mvc3/Scripts/jquery.validate.min.js" type="text/javascript"></script>
    <script src="/Mvc3/Scripts/jquery.unobtrusive-ajax.min.js" type="text/javascript"></script>
    <script src="/Mvc3/Scripts/jquery.validate.unobtrusive.min.js" type="text/javascript"></script>
    <script type="text/javascript">
        function onSuccess() {
            // logged in, carry on
        }
    </script>
</body>
</html>

Analysis

The first obvious thing we can see is that the UnobJS code is slightly lighter, even with strict formatting turned it generally produces less LOC's. But that's not really that important in the grand scheme of things, but is important is the whole unobtrusiveness of the UnobJS source.

In the old MVC2 code look at those inline event handlers at lines 14 and 15, look at that huge code island at line 33, smack bang in the middle of our HTML. It's not wrong per say but it's certainly not best practise. No it's always best to keep your View (HTML) and your Code (JavaScript) separated as much as possible. Imagine if some wayward script was added between line 61 and 62 did something like this

window.mvcClientValidationMetadata = null;
document.forms[0].onclick = function() { };
document.forms[0].onsubmit = function() { };

OHT3HNOES END OF THE WORLD!

Or what if your CDN that served your scripts was down? You are actually going to get JavaScript errors during form submission which can, in various browsers, prevent the form being submitted.

OHT3HNOES END OF THE WORLD AGAIN!

MVC 3/UnobJS on the other hand isn't going to error out and will fall back to a normal form submission - progressive enhancement or just expected behaviour?

On other thing... those inline event handlers pretty much need to have the MS Ajax Library available to work (and not cause errors). What if I was already using jQuery for my project? I'd still need to include MS Ajax on the page even though only generated code required it (well I could write my own API mimicking the required MS Ajax API but why should I?).

Now Unobtrusive JavaScript isn't without it's issues. Using HTML5's data- attributes can invalidate your HTML which can be a showstopper for some projects. One other issue that we may start to see is that EVERYONE is starting to use data- attributes (e.g. jQuery Mobile) I wonder if we are going to start seeing collisions between different libraries?

All in all I don't think there should be an option to turn Unobtrusive JavaScript off, it should just be the only way of doing things :-) but thats just me.

Developing a jQuery Plugin

Fri Oct 22 00:00:00 UTC 2010

Introduction to Ajax

Fri Oct 22 00:00:00 UTC 2010

JavaScript Performance Consideration

Fri Oct 22 00:00:00 UTC 2010

A Brief Introduction to jQuery

Fri Oct 22 00:00:00 UTC 2010

jQuery and YUI: A Comparison

Fri Oct 22 00:00:00 UTC 2010

Practical jQuery Mobile with ASP.NET MVC

Thu Oct 21 23:00:00 UTC 2010

That's a bit of a mouthful. I wanted to write a post about creating a basic jQuery Mobile app but as I started putting the code together MVC became more and more involved so I combined the 2.

The end solution? Lets create a phone directory with 2 main views,

A filterable list of all people with quick info (telephone extension and name) that is grouped and sorted alphabetically, and,
A disclosure view of a selected person showing more details including a photo

[[posterous-content:FddxCHuIgArIuezHEbcn]]

MVC (The Server Side)

jQuery Mobile works by progressive enhancement and uses Ajax to load and parse external links so it has more control over page transitions and Ajax history. This means that we create a plain old website that will work without jQuery Mobile, Ajax or any JavaScript. So I started with the ASP.NET MVC 2 Web Application Visual Studio Template and ripped out everything bar the Home Controller and the 2 views. I also stripped the Site.Master down to the bare bones.

Next I created my model with data access methods,

public class Entry
{
    public string Id { get; set; }
    public string Title { get; set; }
    public string FirstName { get; set; }
    public string Surname { get; set; }
    public string Email { get; set; }
    public string InternalNo { get; set; }
    public string Room { get; set; }
    public string ExternalNo { get; set; }
    public string PhotoLocation { get; set; }

    public static IEnumerable<Entry> GetAll()
    {
        // get all entries
    }

    public static Entry GetById(string id)
    {
        // get a specific entry
    }
}

You can implement your own data access there or just hard code some values. Next I updated the HomeController to return the right models to the views

public ActionResult Index()
{
    return View(from e in Entry.GetAll()
                orderby e.Surname
                group e by e.Surname.Substring(0,1) into g
                select g);
}

public ActionResult About(string id)
{
    return View(Entry.GetById(id));
}

I updated the views to display the information in a straightforward way.

Index.aspx

<%@ Page Language="C#" MasterPageFile="~/Views/Shared/Site.Master" Inherits="System.Web.Mvc.ViewPage<IEnumerable<IGrouping<string,Kas.JQueryMobile.WhosWho.Models.Entry>>>" %>
<%@ Import Namespace="Kainos.JQueryMobile.WhosWho.Models" %>
<asp:Content ID="Content1" ContentPlaceHolderID="TitleContent" runat="server">Whos Who</asp:Content>

<asp:Content ID="Content2" ContentPlaceHolderID="MainContent" runat="server">
    <div>
        <div>   
            <h1>Whos Who</h1>
        </div>
        <div>
            <ul>
                <% foreach (IGrouping<string,Entry> group in Model){%>
                    <li><%=group.Key%></li>
                    <% foreach (Entry item in group){%>
                        <li>
                            <a href="<%=Url.Action("About", "Home", new { id = item.Id }) %>">
                                <%=item.Title %>
                            </a>
                            <p>
                                <strong><%=item.InternalNo %></strong>
                            </p>
                        </li>       
                    <%} %>
                <%} %>                               
            </ul>        
        </div>
    </div>
</asp:Content>

About.aspx

<%@ Page Language="C#" MasterPageFile="~/Views/Shared/Site.Master" Inherits="System.Web.Mvc.ViewPage<Kas.JQueryMobile.WhosWho.Models.Entry>" %>

<asp:Content ID="Content1" ContentPlaceHolderID="TitleContent" runat="server">
    <%=Model.Title%>
</asp:Content>

<asp:Content ID="Content2" ContentPlaceHolderID="MainContent" runat="server">
    <div>
        <div>   
            <h1><%=Model.Title%></h1>
        </div>
        <div style="background: url(<%=Url.Content("~/Images/" + Model.PhotoLocation) %>) no-repeat top right">
            <p><%=Model.Title%></p>
            <p><a href="mailto:<%=Model.Email%>"><%=Model.Email%></a></p>
            <p><%=Model.InternalNo%></p>
            <p><%=Model.ExternalNo%></p>
            <p><%=Model.Room%></p>                       
        </div>
    </div>
</asp:Content>

This leaves us with a normal, not so pretty but fully functioning site.

[[posterous-content:DblJukiumuFlvvqlHokq]]

Now lets inject some mobile niceness into it,

jQuery Mobile

First things first lets put reference to jQuery and jQuery Mobile (script and css) into our site master and update our DOCTYPE to the HTML5 DOCTYPE leaving us with,

<%@ Master Language="C#" Inherits="System.Web.Mvc.ViewMasterPage" %>
<!DOCTYPE html>
<html>
    <head runat="server">
        <title><asp:ContentPlaceHolder ID="TitleContent" runat="server" /></title>    
        <link href="<%=Url.Content("~/ClientBin/jquery.mobile-1.0a1.min.css") %>" rel="stylesheet" type="text/css" />
        <script src="<%=Url.Content("~/ClientBin/jquery-1.4.3.min.js")%>" type="text/javascript"></script>
        <script src="<%=Url.Content("~/ClientBin/jquery.mobile-1.0a1.min.js")%>" type="text/javascript"></script>
    </head>
    <body>
        <asp:ContentPlaceHolder ID="MainContent" runat="server" />
    </body>
</html>

Now we need to tell jQuery Mobile how to layout the pages and mobilise. This isn't necessarily done in script as you might think. jQuery Mobile makes use of HTML5's data- attributes to identify how the page should be laid out and mark areas to specific roles.

Index.aspx

<%@ Page Language="C#" MasterPageFile="~/Views/Shared/Site.Master" Inherits="System.Web.Mvc.ViewPage<IEnumerable<IGrouping<string,Kas.JQueryMobile.WhosWho.Models.Entry>>>" %>
<%@ Import Namespace="Kainos.JQueryMobile.WhosWho.Models" %>
<asp:Content ID="Content1" ContentPlaceHolderID="TitleContent" runat="server">
    Whos Who
</asp:Content>
<asp:Content ID="Content2" ContentPlaceHolderID="MainContent" runat="server">
    <div data-role="page">
        <div data-role="header">   
            <h1>Whos Who</h1>
        </div>
        <div data-role="content">
            <ul data-role="listview" data-filter="true">
                <% foreach (IGrouping<string,Entry> group in Model){%>
                    <li data-role="list-divider"><%=group.Key%></li>
                    <% foreach (Entry item in group){%>
                        <li>
                            <a href="<%=Url.Action("About", "Home", new { id = item.Id }) %>">
                                <%=item.Title %>
                            </a>
                            <p class="ui-li-aside">
                                <strong><%=item.InternalNo %></strong>
                            </p>
                        </li>       
                    <%} %>
                <%} %>                               
            </ul>        
        </div>
    </div>
</asp:Content>

About.aspx

<%@ Page Language="C#" MasterPageFile="~/Views/Shared/Site.Master" Inherits="System.Web.Mvc.ViewPage<Kas.JQueryMobile.WhosWho.Models.Entry>" %>

<asp:Content ID="Content1" ContentPlaceHolderID="TitleContent" runat="server">
    <%=Model.Title%>
</asp:Content>

<asp:Content ID="Content2" ContentPlaceHolderID="MainContent" runat="server">
    <div data-role="page">
        <div data-role="header">   
            <h1><%=Model.Title%></h1>
        </div>
        <div data-role="content" style="background: url(<%=Url.Content("~/Images/" + Model.PhotoLocation) %>) no-repeat top right">
            <p><%=Model.Title%></p>
            <p><a href="mailto:<%=Model.Email%>"><%=Model.Email%></a></p>
            <p><%=Model.InternalNo%></p>
            <p><%=Model.ExternalNo%></p>
            <p><%=Model.Room%></p>
                       
        </div>
    </div>
</asp:Content>

So what have we added here?

data-roles - telling jQuery Mobile what each div actually represents e.g.
- page - a single view (a single html page can have multiple views)
- header - the header of a page
- content - the pages content
- footer - the footer of a page
- list-view - a special role to specifying that the content is a list
- list-divider - a divide for a list that doesn't do anything but look different
data-filter - telling jQuery Mobile that it should provide filtering on this view
ui-li-aside class identifies that this is some aside information for this list item

And that is it. We have taken a static site and without writing any code (bar markup) created a mobile app. Probably might be a good idea to include an application cache manifest file as well so there is some semblance of offline capability provided. But that's for another day.

[[posterous-content:pJHwwsHHmrfAkJIJFkrp]]

ASP.NET MVC3 JsonValueProviderFactory

Wed Oct 20 23:00:00 UTC 2010

ASP.NET MVC 3

Anyone who's been involved in an ASP.NET MVC project that is quite Ajax heavy will probably have noticed that something was always missing. Imagine this front end scenario,

<% using (Html.BeginForm()) { %>
    <p>
        <%=Html.LabelFor(m => m.Username)%>
        <%=Html.TextBoxFor(m => m.Username)%>
    </p>
    <p>
        <%=Html.LabelFor(m => m.Password)%>
        <%=Html.TextBoxFor(m => m.Password)%>
    </p>
<%} %>
<script type="text/javascript">
    $("form").submit(function (evt) {   
        // extract values to submit         
        var form = $(this),
            username = form.find("[name=Username]").val(),
            password = form.find("[name=Password]").val(),
            json = JSON.stringify({
                Username: username,
                Password: password
            });

        $.ajax({
            url: form.attr("action"),
            type: 'POST',
            contentType: 'application/json; charset=utf-8',                
            dataType: 'json',
            data: json,                
            success: handleLogin
        });

        // stop form submitting
        evt.preventDefault();
    });
</script>

Which posts to the following action

[HttpPost]
public ActionResult Index(LoginModel model)
{
    // do login
    return Json(new 
    {
        Success = true
    });
}

We have a login screen that is submitted via ajax. Now this is quite a contrived example (ideally you'd be performing a normal post via ajax in this situation) but there are many instances where this sort practise would apply (ExtJS' RESTful DataWriters for example).

MVC 2

In MVC 2 this wouldn't work immediately. The default model binder in MVC 2 uses Request parameters to bind to model properties but in this case there are none as the ajax content is the body of the request.

To accommodate this sort of request in MVC 2 we had to provide a custom model binder that knows how to deal with JSON requests,

public class JsonModelBinder : IModelBinder
{
    public object BindModel(ControllerContext controllerContext, ModelBindingContext bindingContext)
    {
        HttpRequestBase request = controllerContext.HttpContext.Request;
        string jsonStringData = new StreamReader(request.InputStream).ReadToEnd();
        return JsonConvert.DeserializeObject(jsonStringData, bindingContext.ModelType);
    }
}

And attribute our actions model argument telling it to use this binder,

[HttpPost]
public ActionResult Index([ModelBinder(typeof(JsonModelBinder))]LoginModel model)
{
    // ...
}

It'll do the job but it's incredibly messy.

MVC 3

MCV3 fills this gap thanks to the JsonValueProviderFactory. The JVPF operates at a higher level than a model binder. Basically what it does when a JSON request is received is that it pulls the values out of the JSON body as key value pairs which means they are available to the model binders including the default model binder. No special wiring required, no custom model binders (unless of course you want one) just out-of-the-box workingness!

[HttpPost]
public ActionResult Index(LoginModel model)
{
    // do login
    return Json(new 
    {
        Success = true
    });
}

I know most of this post was taken up by MVC2 specific implementation but isn't that really the point? MVC3 is a nice refinement of MVC2 there isn't anything new exactly but the core stuff that is there has been made easier and more configurable.

Client Side: Processing vs Responsiveness

Mon Oct 18 23:00:00 UTC 2010

The examples below are probably best run in a browser that has a relatively slow script engine (IE for example)

One of the problem with the whole Web2.0/Rich Client concepts is that people seem to think it's OK to shove processing that should be done on the server on to the client. This is never the answer - you can control resources on the sever but never on some random browser half way across the world/country/street/room. There are, however, situations where client side processing is going to be mandatory (think Complex Grids and Calculations etc). This obviously leads to a loss of responsiveness on the client (to varying degrees). One of the big problem with large processing is that it obviously takes time. Imagine this scenario...

function doProcessing(){
 
  /* get the progress indicator element */
  var el = document.getElementById("progress-meter");
 
  /* update element */
  el.innerHTML = "Processing";
 
  /* do long running processing */
  for(var i = 0 i &lt; 999999; i++){
    var j = Math.sqrt(i);
  }
 
  /* update element */
  el.innerHTML = "Done!";
}
 
doProcessing()

You'd expect that this code updates the "progress-meter" element with "Processing", does some long computation and then updates the "progress-meter" again to indicate it's done. But it doesn't. Screen repaints/updates do not happen until calls complete. So what is actually happening? Well the expected outcome DOES happen it's just both updates happen after the call completes so the first update occurs so fast you never see it. See It In Action. You'll also notice that the entire screen is locked while this processing occurs and that just leads to a bad user experience. So what can be done?

Use of setTimeout()

Javascript is single threaded which leads to the issues above but there is a way create pseudo-threading using setTimeout. The setTimeout function executes code after a set time in milliseconds (what a surprise!). So delaying this execution (even by 0 miilliseconds) allows screen updates to occur before the processing begins. This requires a small change to the code.

function doProcessing(){
  var el = document.getElementById("progress-meter");
  el.innerHTML = "Processing";
  setTimeout(function(){
    for(var i = 0; i &lt; 9999999; i++){
      var j = Math.round(Math.sqrt(i));
    }
    el.innerHTML = "Done!";
  },0);
}
 
doProcessing()

So if you give this a go you will see there is a much better feedback response compared to the first. This approach solves our initial problem but we still have the issue that the screen is locking up. At least now the users knows something is happening. Another issue this creates is that some browsers prompt users if scripts take to long allowing the user to cancel potentially important tasks.

[[posterous-content:jcdDmwFBuEneeneritwd]]

Asynchronous Processing

Now we are getting into the good stuff. To prevent the screen locking up during any complex processing activity it is possible to apply the setTimeout technique but only over a smaller subset of the processing activity each time. This will achieve a sort of asynchronous effect allowing the user to do other stuff while the processing happens in the background. This is a basic example,

function complete(){
  alert("Processing Complete");
}
 
function doProcessing(callback){
 
  var el = document.getElementById("progress-meter");
  el.innerHTML = "Processing";
 
  /* setup iteration */
  var iterations = 9999999;
  var chunks = iterations/100;  /* each chunk is 1% of the overal processing count*/
  var i=0;
 
  /* self executing anonymous function */
  (function() {
 
    /* process chunk */
    for(var count = 0;i&lt;iterations;i++) {
      var j = Math.round(Math.sqrt(i));
      count++;
      if(count == 99999)
        break;
    }
 
    /* more to process */
    if(i&lt;iterations){
      /* update screen */
      el.innerHTML = Math.round((i/iterations)*100) + '% Complete';
 
      /* recurse for next segment */
      setTimeout(arguments.callee,0);
    }else{
      /* update screen */
      el.innerHTML = "Done!";
 
      /* call optional callback */
      if(callback){ callback() }
    }
  })();
}

doProcessing(complete);

You can see this in action for yourself. If you run the example you will see that there is a constant update (after each chunk is processed) and that screen control is still available. So this has solved both our issues but has introduced new considerations.

Executes asynchronously therefore we need to support a callback mechanism as it's not procedural (see code)
Processing time increases (see below)

You will see that the length of time it take to process the complete task is a lot longer than the other 2 examples.. this is due to the increase in complexity of the code (recursion, another function call etc) and this is the trade off. Processing time is inversely proportional to the size of each processing chunk

[[posterous-content:ImAAruEoruBBGBGeBetI]]

Chunk Size (y) vs Response Time (x)

As the chunk sizes get smaller you get a more responsive UI but the processing time can get scarily long. The example chunks at about 1% of the total task size and that seems to be a good balance (for this task at least) between responsiveness and processing time. I threw together a little suite of yielding techniques with various update times etc and you can play around with that here

< HEAD >ache: Including JavaScript in ASP.NET Master Pages

Mon Oct 18 23:00:00 UTC 2010

Problem

Include a Javascript file in the head of of an ASP.NET WebForms Master Page whose src attribute should contain an absolute (from Application Root) path e.g.http://localhost:8080/js/jquery-1.3.2.js (single application server) or http://localhost:8080/MyProject/js/jquery-1.3.2.js (multi application server). Sounds simple enough but it's surprisingly difficult.

Attempt 1: Use a fixed path. (Partial Success)

<script src=?/js/jquery-1.3.2.js? type="text/javascript"></script>
<script src="/NIApp/js/jquery-1.3.2.js" type="text/javascript"></script>

While this "works" it isn't exactly the most resilient to server changes - what if we move to/from a single/multi application server? We need to re-write all our URL's

Attempt 2: Use Application Root Reference (~) (Fail)

<script src=?~/js/jquery-1.3.2.js? type="text/javascript"></script>

This just plain doesn't work because script tags do not get parsed by ASP page renderer so the URL doesn't get converted and what you see here is what you end up with on the page which obviously isn't correct.

Attempt 3: Use Server Code (<%= %>) (Partial Success/Fail)

<script src=?<%=Page.ResolveUrl(?~/js/jquery-1.3.2.js?)%>? type=?text/javascript?></script>

Again this is going to fail in certain situations (but not all). This will fail when your head tag has a runat="server" attribute present. In some cases you can simply remove the attribute but if you are using the "out-of-the-box" CSS Themes/Skins you NEED this.

Attempt 4: Use a Second <HEAD> Element (Partial Success)

<head> 
    <script src=?<%=Page.ResolveUrl(?~/js/jquery-1.3.2.js?)%>? type=?text/javascript?></script> 
</head> 
<head runat=?server?> ? </head>

This will work as you would expect - the only problem is that it isn't valid Markup and will fail if you throw it through W3C Validators.

Attempt 5: Add script tags "Outside" Head Element (Partial Success)

<head runat=?server?> ? </head> 
<body> 
    <script src=?<%=Page.ResolveUrl(?~/js/jquery-1.3.2.js?)%>? type=?text/javascript?></script> 
</body>

Again this will work the only problem is you can never guarantee another developer won't stick code (which is dependant on your global script) BEFORE your inclusion of said script which is a bit of a <head>ache (i.e. in the <head> element - get it?).

Attempt 6: Insert Script Tags via MasterPage Page_Load Event (Success)

protected void Page_Load(object sender, EventArgs e)
{
    HtmlGenericControl script = new HtmlGenericControl('script');
    script.Attributes.Add('type', 'text/javascript');
    script.Attributes.Add('src', ResolveUrl('~/js/jquery-1.3.2.js'));
    Page.Header.Controls.AddAt(0, script);
}

I suppose this is the first real successful attempt. It does what it says by dynamically creating new Script tag element and inserting them to the top of <head>. I inserted everything at the top of the head so no matter where people put their scripts it's going to have all global resources available. The only problem with this is that EVERY Page_Load event is going to dynamically inject new Elements into the head and it's not exactly a very clean separation of mark-up and code.

Attempt 7/Solution 1: Use Data Binding to Evaluate src Attribute (AKA The Holmes Technique)

ASPX

<head runat="server">    
    <script type="text/javascript" src=?<%# ResolveUrl("~/js/jquery-1.3.2.js")%>?>   </script>
    <asp:ContentPlaceHolder ID="head" runat="server"></asp:ContentPlaceHolder>
</head>

Jonny Holmes uncovered a more elegant solution to this issue and it is described above. It gets around the issue of not being able to use <%= %> tags by using a Data Binding Expression. And all you have to do is perform a DataBind() during the Page_Load event and you're sorted like Ebenezer.

protected void Page_Load(object sender, EventArgs e)
{
    //Called for the javascript references in the header of the master page
    Page.Header.DataBind();
}

Anyone got any other methods to do this?

JavaScript Proxy Pattern

Mon Oct 18 23:00:00 UTC 2010

More often than not when using 3rd party Javascript widgets/plugins/frameworks is that they don't behave in exactly the way that you want forcing you to change tactics, write some horribly messy boilerplate code, hack the original code or look for alternatives. The proxy pattern is a useful pattern that provides a sort of AOP-lite technique that can be used to transparently manage 3rd party code without having to make breaking changes or repeating horrible code from a single point in the code.

Functions are 1st class objects in JavaScript and this is the feature that powers this pattern. Imagine the setup below,

// AJAX Object
var XHR = {
    get : function(url, callback){
        // perform ajax call to url
        // call the callbakc with the returned data
    }
}
 
// specific internal ajax functions.  
// These are 3rd party code that shouldn't be touched
function GetData(url, callback){
    XHR.get(url, function(data){
        var parsedData = parse(data);
        callback(parsedData);
    });
}
 
function GetData2(url, callback){ /* similar thing to GetData*/ }
...
function GetDataN(url, callback){ /* similar thing to GetData*/ }

Now imagine there are a set of URLs that require some sort of authentication token passed along with the request. How do we deal with that? Well some people would just hack the internal functions but that's a bad idea as it probably breaks other parts of the system. Another way to do it would be to write your own Ajax framework for the new calls you want to make but that's overkill. The proxy pattern can help here.

// THE PROXY BLOCK
(function(){
 
    // create empty list of secured URLs
    XHR.securedURLs = [];
 
    // point to original function
    var _get = XHR.get;
 
    // create proxy function
    XHR.get = function(url, callback){
        // is url a secured url?
        if(XHR.securedURLs.indexOf(url) &gt; -1){
            // append authentication token
            url += "?authToken=" + XHR.token;
        }
 
        // call original get function with potentially updated url
        _get(url, callback);
    }
})();
 
XHR.token = "AUTH_TOKEN";
XHR.securedURLs.push("https://secured.kainos.com");
 
GetData("https://secured.kainos.com", function(d){ /* code */ } );  // via proxy
GetData("http://notsecured.kainos.com", function(d){ /* code */ } );  // via original

So basically if the URL is a secured URL (present in the XHR.securedURLs array) then there will be a bit of pre-processing done to the URL before the function is called. Otherwise the normal function is called. That is the primary aim of the proxy pattern - to enhance existing functionality - at the very least the proxied function should fall back to the original function otherwise other parts of the system could be come affected (unless you know what you are doing of course - cancelling illegal calls etc).

Webforms & jQuery: A Comparison

Sun Oct 17 23:00:00 UTC 2010

This one has been pulled from the archives but I still think it is quite relevant

I tend to be a bit picky when it comes to seeing what is generated by server-side technologies. The first thing I do when I write a page (JSP, ASPX – whatever) that uses some sort of tag library or user control is check the resultant HTML/CSS/Javascript. When it comes to ASPX's this has always been a major issue for me – the code it generates, the JS it pulls down, the amount of redundant traffic, the actual quality of the code – all of it annoys me. Just out of curiosity I wanted to do a “contrast and compare” study to see the differences between the DIY approach and using Webforms…

The use case for the study is simple

A simple screen with a star rating (1-5)
User selects a rating by clicking a star
The result is posted back, via ajax, to the server
The front end is updated with message indicating the selection and the rating system is disabled

Technologies Used

Webforms

jQuery

ASP.NET ScriptManager Control

ASP.NET Update Panel Control

ASP.NET Ajax Toolkit Star Rating Control

ASP.NET Label Control

ASP.NET MVC Framework

jQuery v1.3.2

jQuery Star Rating

There is not much really to draw from this table except for Webforms the majority of the technology is server-side whereas the jQuery stuff is more client side focused. This is one of the big arguments that people use when arguing about webforms. It's all server side work which reduces client side scripting – but that's a bit blind. It's server-side for the developer but the end code isn't any less client side (it's simply auto-generated).

Development Effort Required

I'm going to end up being biased on this. I am 10,000 times more familiar with the MVC paradigm and jQuery than I am with Webforms. Ultimately it took me about the same amount of time to knock up both applications. But here are some points to note.

Webforms

Drag and drop of all the controls to knock up a simple page. The bulk of the work came from my lack of understanding I guess. I initially stuck the Star Rating control on the page and set up the CSS Classes, this failed as I didn't have the ScriptManager control added to the page. The next issue I faced was that my server side code wasn't actually having any effect – I had to turn AutoPostBack on for it to work but this wasn't Ajax-y. I got around this by wrapping the code in an UpdatePanel.

[[posterous-content:bBdcwIItewquatsyfJyv]]

Default.aspx

[[posterous-content:vfJuAxpqzbFsFoFBFwAE]]

Default.aspx.cs (Code Behind)

jQuery

I used ASP.NET MVC framework to generate my empty page, and controller. I then added a simple one liner Action to my controller to handle the Ajax response. I included jQuery and Star Rating (JS & CSS) to my page and 5 radio buttons that act as the star rating (automatic progressive enhancement) and wrote the initializer code for the star rating stuff. For me that was it – completely straight forward with little configuration.

[[posterous-content:xqpnygHdsdJiruvDqkpf]]

Index.aspx

[[posterous-content:kgpqwhjHGDbiqHJCCvCA]]

RatingController.cs

Resultant Page Weight & Render Time

Webforms

[[posterous-content:uCAmokcAGoyHstBqGxsj]]

Page Weight

[[posterous-content:nDxjAJoJDbwurDtdAnvx]]

Render Time

jQuery

[[posterous-content:cHfxHIJCwxwiGDCgCIuw]]

Page Weight

[[posterous-content:zsChEvClgasifCgquBpo]]

Render Time

Now the real differences are starting to show. The jQuery Solution is light years ahead when it comes to page weight (with marginally faster render times). It's also worth noting here that the Webform solution automatically used GZIP compression for it's scripts whereas the MVC didn't so the results are actually skewed in favour of Webforms (not that it makes a difference).

Here in lies one of my issues with the Webform solution – it pulls down the massive ASP.NET JavaScript framework even though it doesn't use 99.9% of it. jQuery is a complete framework but it's ethos is that it gives you the bare minimum set of tools to get the job done. Look at my code (both server side and client side) – is it at all complex? Not really.

One argument I guess you could use here is that as the page grows the Webform solution, because it has most of the scripts it needs, shouldn't grow much bigger, where as the jQuery solution will. I haven't tested this so I could be wrong but it's a valid argument. That said given any situation I'd be willing to bet I could at the very least match page weight and render time against any Webform solution using jQuery (and MVC).

Server Responses

Webforms

[[posterous-content:rChlEcegkzzzEgEkxiAc]]

Request Parameters

[[posterous-content:gIgansGldxpvrcfvkapa]]

Response Body

jQuery

Request Parameters

Response Body

Again there is a big difference here. Because a lot of stuff is automated on the Webform side there is a lot of configuration (Viewstate etc) that needs to be passed about. Even for this simple solution the overhead is considerable – Firefox is giving me traffic (response only sans headers) of 1 byte for jQuery and 3k for Webforms – now multiply that by 2 to roughly include the request as well.

Which is easier to understand here?

Conclusion

A lot of the arguments and observations brought up here aren't purely Webform vs jQuery related – a lot have to do with Client vs Server programming but they are still valid. I admit I have a firm grasp of client side coding so the choice for me is obvious but the results here speak for themselves – there is a considerable overhead involved in the Webform world that must be accepted if you want to be able to drag and drop controls and save possibly a small amount of development time.

I am in no way a veteran Webform developer and I am sure there are folks out there that could strip my example down to the bare minimum and get similar results and to them I open floor – bring it – I haven't begun!

Strong Typing vs. Magic Strings (Part II)

Sun Oct 17 23:00:00 UTC 2010

One of the problems with ASP.NET MVC is it's use of magic strings in almost every facet of the architecture. I've mentioned this before but I have a few other things I'd like to mention. In my last post I used T4MVC to generate static classes that layer on top of the magic strings but if these don't generate when you want them to they become out of sync and break compile time checking. One solution is to generate the "proxy" files one every save however if you have a large project this could be a bit of a pain.

Other solutions exist already. MVC2 comes with strongly typed HtmlHelpers so which turns

<%=Html.TextBox("FullName")%>

Into,

<%=Html.TextBoxFor(m => m.FullName)%>

Great - compile time checking and the ability to reflect things such as attribute for generating validation etc. Problem is they didn't really do much else. Controller methods still suffered from Magicstringitis. RedirectToAction still needed magic strings for controller and action names and ModelState.AddModelError still needed string for property names.

MvcContrib

MvcContrib offers a load of non-official but excellent extensions to MVC and one of the things that comes bundled with the build is a strongly typed RedirectToAction extension method. This lets change this,

this.RedirectToAction("Index", "Home");

Into

this.RedirectToAction<HomeController>(c => c.Index());

Yet again - compile time checking, refactoring support and no magic strings - Win. But that leaves one last thing that still bugs me - ModelState.AddModelError. To add an error for a particualr property of your model you still need to do something like this...

ModelState.AddModelError("Username", "Username is already in use")

YAMS! Yet another magic string!

DIY

Seeing as I couldn't find a solution from my good friend Google I decided to see if I could do it myself. Turns out it's bloody easy. This extension method will allow us to get rid of magic strings when expecting to map to a property of a simple class (such as a model or DTO)

/// <summary>
/// Adds a model error using strongly typed lambda expressions
/// </summary>
public static void AddModelError<TModel>(
    this ModelStateDictionary modelState, 
    Expression<Func<TModel, object>> method, 
    string message)
{
    if (method == null)
    {
        throw new ArgumentNullException("method");
    }
 
    MemberExpression mce = method.Body as MemberExpression;
    string property = mce.Member.Name;
    modelState.AddModelError(property, message);
}

And you can now call it like this...

ModelState.AddModelError<Model.User>(u => u.Username, "Username already in use");

Neat!

One more bit of proof that lambda expressions are awesome - true there is reflection involved so it's going to be slower than magic strings but in my daily use I haven't really noticed any performance hits so (thanks to Jeff Atwood @ codinghorror.com) I can give it a "Works on my Machine" badge.

Multi-Touch Reactive Extensions

Sun Oct 17 23:00:00 UTC 2010

Link: Reactive Extensions for .NET

It's been a while since I've played with MultiTouch functionality so I decided it was time to dust off the old TouchSmart and put it to good use. The purpose wasn't to play with MultiTouch functionality but more to see how we could simplify managing isolated events using RX. The MultiTouch API is good for this because by its very nature MultiTouch requires us to collect multiple events simultaneously and wire them up to create gestures. Unfortunately the current iteration of MultiTouch API wrappers for Silverlight and WPF (touch, MIRIA etc) are quite basic and when working with custom gestures you are forced to introduce a lot of global code such as flags etc. Not pretty!

Imagine the following scenario.

2 touch points on screen (let call them HoldRight and HoldLeft). Both expose 2 events Hold and Release
When a person presses and hold a finger over BOTH points trigger an event
If only one is being pressed and held do not fire the event
If the user lifts their finger or fingers off the points and repeats the process the event should fire again.

This is a fairly simple example but actually wiring this up requires the use of nasty global variables and a bit of repetition. Here is a very quick implementation of this.

public partial class MainPage : UserControl
{
    private bool _rightHeld = false;
    private bool _leftHeld = false;
 
    public MainPage()
    {
        HoldLeft.Hold += HoldLeft_Hold;
        HoldRight.Hold += HoldRight_Hold;
        HoldLeft.Release += HoldLeft_Release;
        HoldRight.Release += HoldRight_Release;
    }
 
    private void HoldRight_Release(object sender)
    {
        _rightHeld = false;
        DoCheck();
    }
 
    private void HoldLeft_Release(object sender)
    {
        _leftHeld = false;
        DoCheck();
    }
 
    private void HoldRight_Hold(
        object sender, 
        GestureHoldEventArgs e)
    {
        _rightHeld = true;
        DoCheck();
    }
 
    private void HoldLeft_Hold(
        object sender, 
        GestureHoldEventArgs e)
    {
        _leftHeld = true;
        DoCheck();
    }
 
    private void DoCheck()
    {
        if (_rightHeld && _leftHeld)
        {
            // Fire the event
        }
    }
}

As you can see I have made use of 2 global variables a DoCheck method that checks if the 2 canvases are currently in a "held" state and fires the event if they are. All the event handlers do nearly the same thing. Sure I could refactor this to use maybe 2 event handlers and inspect the sender but that starts getting messy.

Reactive Extensions to the Rescue!

Reactive extensions let us "compose" events so we can pass them around and filter them like first class citizens. Because of this we can actually create custom events by combining. Tackling the same issue as above we can achieve the same functionality without the need for dodgy global vars and boilerplate code. Lets look at the code first.

public partial class MainPage : UserControl
{
    public MainPage()
    {
        // compose events
        var leftHold = Observable.FromEvent<GestureHoldEventArgs>(
            HoldLeft, "Hold");
        var rightHold = Observable.FromEvent<GestureHoldEventArgs>(
            HoldRight, "Hold");
        var rightRelease = ObservableEx.FromMultiTouchReleaseEvent(HoldRight);
        var leftRelease = ObservableEx.FromMultiTouchReleaseEvent(HoldLeft);
 
        // subscribe to dual hold event
        leftHold.Zip(rightHold, (l, r) => l)
            // listen until either release triggered
            .TakeUntil(leftRelease.Amb(rightRelease))
            // trigger the event
            .Subscribe(_ => { /* trigger event */ });
    }
}
 
public static class ObservableEx
{
    public static IObservable<IEvent<EventArgs>> FromMultiTouchReleaseEvent(
        TCanvas canvas) {
        return Observable.FromEvent<TCanvas.ReleaseHandler, EventArgs>(
                h => new TCanvas.ReleaseHandler(e => { }),
                h => canvas.Release += h,
                h => canvas.Release += h
            );
    }
}

First things first for some random reason the implementers of MIRIA decided to create en event the actually passed NO event args - this is not normal and could easily be considered and anti-pattern. Because of this I had to use the long winded Observable.FromEvent which is created as a helper method.

So what are we doing here?

Compose our events into first class citizens - Observables
Zip left and right hold events. Zipping basically combines 2 Observables and publishes or emits a value when both observables have emitted a value. The second argument in Zip is the transform function that converts the 2 emitted values into 1 value. We don't care about this value so just return some arbitrary value.
Take the Zip generated observable and keep publishing it's values until Either the left or right release events are triggers (Amb = most ambitious - publishes first value to appear)
Subscribe to this super composed event. When this happens we can fire our event

What actually happens on the front end is irrelevant (the solution looks like this and will spin when the two Thumb areas are pressed and held)

[[posterous-content:ECxarBiHmvIvJutnbJaJ]]

But the fact we have managed to combine 4 isolated events without having to use boilerplate is very nice. This gives us a lot of power to create and control complex gestures - not limited to MultiTouch but any sort of UI interactions (Mouse events, Web service calls etc).

* I'll keep saying this. RX doesn't do anything new or solve any unsolvable problems but what it does do is allow us to do things in a neater way. Now there is a bit of learning curve in it - you really need to start thinking in RX but once you wrap your head around RX is a nice tool to have on your tool belt. No it's not complexity for complexity's sake - I honestly believe the code above, when used in the real world, will hep reduce complexity and make maintenance easier.

Good Fortune Awaits!

T4MVC: Strong Typing vs Magic Strings

Sun Oct 17 23:00:00 UTC 2010

Magic strings are everywhere in ASP.NET MVC 2. Less so than in version 1 and some of the Beta and RC releases but there are still some kicking around. The problem with literals is that they don't give you any compile time error checking or refactoring ability. Say for example you have an action link that points you to the Edit action of the Person controller - you write it like this.

<%=Html.ActionLink("Edit", "Person", new { id = 12 }) %>

Now image you've written something like this over numerous pages and all of sudden you change the name of the Action or the Controller. These are literal strings so you can't use the refactor tools in VS or get any compile time error when you build. The only real way to ensure everything has been changed would be to do a quick regression test. Nightmare.

T4MVC

T4MVC makes use of the Visual Studio bundled T4 Text Template Transform Toolkit (Template based code generation tool) to generate invisible classes that give you strongly typed links to Controllers, Actions, Views and even JavaScript, CSS, Images and other static links.

T4MVC is part of the MVCContrib project (though completely standalone). You simply download the zip with the 2 T4 files and add them to the root of your MVC project and you are done. Here are some examples of it in action.

VIEW NAMES

<% Html.RenderPartial("DinnerForm"); %>
<% Html.RenderPartial(MVC.Dinners.Views.DinnerForm); %>

And within a controller

return View("InvalidOwner");
return View(Views.InvalidOwner);

CONTROLLER ACTIONS

<%= Html.ActionLink("Delete Dinner", "Delete", "Dinners", new { id = Model.DinnerID }, null)%>
<%= Html.ActionLink("Delete Dinner", MVC.Dinners.Delete(Model.DinnerID))%>

STATIC FILES

You also get the ability to reference static files so that if they are moved or renamed you'll not be missing images etc. - something that is a pain to test most times.

<img src="/Content/nerd.jpg" />
<img src="<%= Links.Content.nerd_jpg %>" />

Another nicety is that you also get this for JavaScript files but without having to do anything it will swap out debug (uncompressed) versions of you files for minified versions when you move into production. As long as you have <filename>.js and <filename>-min.js side by side it will determine which one to use based on the build environment.

<script src="/Scripts/Map.js" type="text/javascript"></script>
<script src="<%= Links.Scripts.Map_js %>" type="text/javascript"></script>

Summing Up

T4MVC is a nice tool to get around magic strings. There are other things out there that give "fluent" syntax to methods that use literals which use delegates instead of generating any sort of proxy classes. Which is better is a question I still can't answer confidently. I have had some minor issues using T4MVC (mostly my fault) and I am a bit dubious about the syntax for static files but it's a decent starter for 10 and doesn't need any thought to implement and use. Of course you could also roll own specific T4 template (or extend T4MVC) but that obviously requires a bit more work.

jQuery Mobile Quick Look

Sun Oct 17 23:00:00 UTC 2010

jQuery Mobile was released over the weekend or at least the alpha version of it was (full release scheduled for some time on 2011).

So what is it like? Well first things first it's early days, there are certainly more than few rough edges but given the development timeline (~2 months) it is quite an impressive effort put in by the Filament Group and others.

iThings

The first question most people will ask (or the first thing they will try and do) will probably be

So how does it look on the various iThings (iPad, iPhone, iPhone 4)?

Well it looks good for the most part. There are some issues around page transitions and displaying dialogs (especially on the iPad) - things like page flickering, hidden content becoming visible after the page has loaded etc. and while they are a bit annoying they don't "break" anything.

As the framework isn't targeting iOS specifically there are some behaviours that would wouldn't necessarily expect if you used this as a native app (using PhoneGap or whatever). Toolbars aren't explicitly fixed and when they are fixed disappear on scroll and repositioned after scrolling has finished. Select boxes don't display the normal pickers found in native iOS but instead use a custom dialog for selection. Again these aren't criticisms, they work well, but I have heard people already complaining because they expect the framework to produce a fully native looking app.

Sencha Touch

When comparing frameworks many peoples first thoughts will be

How does it compare to Sencha Touch

To be honest the comparison is like comparing apples to oranges, or jQueryUI to ExtJS. Both projects have also different goals. While Sencha currently aims to provide a rich native like experience on a small subset of mobile browsers (specifically iOS and Android), jQuery Mobile aims to create a consistent, mobile/touch capable experience across a much wider range of mobile browsers. This is by no means easy. An interaction style one device may not be suitable for another.

FOUC

One of the goals of jQuery Mobile is to progressively enhance the base page to become more mobile capable. Because of this the base page is very basic with jQuery Mobile specific stylers applied when the DOM is ready. This can lead to some initial FOUC (Flash of Unstyled Content). We can get around this by applying some of the basic jQuery Mobile styles to begin with and I suspect we will see this tactic being used on future releases.

[[posterous-content:EeHJJwvCqFJrzcdxeEEb]]

Accessibility

One thing I never really though of in a Mobile Framework is accessibility. This is just short sightedness on my part as it is as relevant in the mobile world as it is in the desktop world. jQuery Mobile aiming to be fully accessible through it's use of ARIA roles and progressive enhancement.

The Technicals

From a technical/developer stand point jQuery Mobile is quite nice. Lets look at a few of the highlights,

Pages

Sites can be built as a single page with sub sections being marked as "pages" that jQuery Mobile can navigate. External pages are load using ajax (rather than simply navigating to them), this give jQuery Mobile better control over transitions, and history management. A simple page looks like this,

<!DOCTYPE html> 
<html> 
    <head> 
        <meta charset="UTF-8" />
        <title>jQuery Mobile</title> 
        <link href="Styles/jquery.mobile-1.0a1.css" rel="stylesheet" type="text/css" />
        <script src="Scripts/jquery-1.4.3.js" type="text/javascript"></script>
        <script src="Scripts/jquery.mobile-1.0a1.js" type="text/javascript"></script>
    </head> 
    <body> 
        <div data-role="page">
            <div data-role="header">   
                <h1>Page 1</h1>
                <a href="#about">Page 2</a>
            </div>
            <div data-role="content">Page 1 Content</div>
        </div>
        <div data-role="page" id="about">
            <div data-role="header">   
                <h1>Page 2</h1>
            </div>
            <div data-role="content">Page 2 Content</div>
            <div data-role="footer">Page 2 Footer</div>
        </div>
    </body>
</html>

So lets break this down and pick out the interesting parts,

<!DOCTYPE html>

HTML5 DocType which older browsers can handle gracefully

<link href="Styles/jquery.mobile-1.0a1.css" rel="stylesheet" type="text/css" />
<script src="Scripts/jquery-1.4.3.js" type="text/javascript"></script>
<script src="Scripts/jquery.mobile-1.0a1.js" type="text/javascript"></script>

Scripts and styles included. Now from a mobile perspective this is still quite a bit of script especially over some flaky 3G connections but in comparison to some of the frameworks out there it is still quite small.

<div data-role="page">
    <div data-role="header">   
        <h1>Page 1</h1>
        <a href="#about">Page 2</a>
    </div>
    <div data-role="content">Page 1 Content</div>
</div>
<div data-role="page" id="about">
    <div data-role="header">   
        <h1>Page 2</h1>
    </div>
    <div data-role="content">Page 2 Content</div>
    <div data-role="footer">Page 2 Footer</div>
</div>

This section describes 2 pages. A page is marked as a page by its `data-role` attribute and can contain 3 elements marking a header, the page content and the footer. Unless stipulated the first page will be first page in the DOM. Navigation between pages is handled using location.hash changes (page 1s button links to Page 2 in the example).

Themeing

Themeroller will see an overhaul on the official release which will allow you to create custom themes a la jQueryUI. Currently jQuery Mobile comes with 2 themes both of which have a number of swtaches. This is intended to create contrast between UI elements (e.g. for marking mandatory fields, or highlight primary buttons on a form etc.)

Dialogs

Dialogs in jQuery Mobile are simply pages with a `data-rel="dialog"` added to them so the same rules apply as above.

Form Elements

Forms in jQuery Mobile are plain old HTML(5) forms which is actually a VERY nice feature. Sencha/ExtJS forms are nice but when using SELECTs etc. can become a real pain to manage. jQuery Mobile uses a much lighter approach. Some screenshots (iPhone 4)

[[posterous-content:qipBwFyayBhuztrHFpJI]] So thats a very quick overview of jQuery Mobile. I encourage you to play around with and get those bugs filed. As I've said it's early days but it's a big goal they are aiming for and they are currently well on track to produce a very nice framework come the official release.

jQuery 1.4.3 & jQuery Mobile Released

Fri Oct 15 23:00:00 UTC 2010

As expected jQuery 1.4.3 and jQuery Mobile (alpha) have dropped today. The official announcements are here

Having had the chance to play around with these already here are a few of the highlights that I think are worth mentioning. This first post will talk about jQuery Core as I need a bit more time with jQuery Mobile to get a better feel for it.

jQuery 1.4.3

.data()

One of the first cool things that 1.4.3 brings with it is a tweak to the .data() method to allow it to work with not only DOM objects but plain old JavaScript objects as well. So what? Well it opens the doors for the recently announced Data Linking plugin from Microsoft (blogged about it before) until now ran on a patched version of jQuery Core. Essentially this give us the ability to perform data binding between UI elements and a model object. This will be familiar to people who use binding in Silverlight. This is all made possible via the newly introduced changeData event that will get fired when .data() is used to modify a data value.

var user = {
    username: 'jameshu',
    fullName: 'James Hughes'
};

$(user).bind("changeData", function( event, name, value ) {
  console.log(name + " changed to " + value);
});

$(user).data("username", "jameshu2");

// => username changed to jameshu2

Another nice new feature rolled into .data() is that all HTML data attributes are automatically parsed as data on initial selection of the element(s). This behaviour is similar to the metadata plugin that has been around for some time. In fact data values are coerced into the applicable types ("true" becomes Boolean, "{o:1}" becomes a JavaScript object etc.). This is a nice feature to have but I wonder how resilient it is to weird strings.

So for,

<div data-role="container" data-hidden="true" data-options='{"name":"James Hughes"}'></div>

The following will be true

$("div").data("role") === "container";
$("div").data("hidden") === true;
$("div").data("options").name === "James Hughes";

This is going to play very nice with the Unobtrusive JavaScript/Ajax features of the ASP.NET MVC3 (which already employs something similar)

Modularity

One of the issues many people had with jQuery was it was "all or nothing". This was fine at the start but as the file size started to grow people felt that they should be able to only use a small part of jQuery without having to pull the whole thing down. This release sees a fairly big refactoring to try and make this wish a reality. jQuery Core has been broken up into a number of sub modules which gives us the ability to load modules on demand (see Modernizr and Require.JS) or simply use the modules we need.

jQuery.type()

There are plenty of little cross browser differences in the typeof operator and as such it's not always reliable. The jQuery type method abstracts out all those little quirks to correctly determine arrays etc.

jQuery.type(true) === 'boolean'
jQuery.type(3) === 'number'
jQuery.type('test') === 'string'
jQuery.type(function(){}) === 'function'
jQuery.type([]) === 'array'
jQuery.type(new Date()) === 'date'
jQuery.type(/test/) === 'regexp'

And The Rest

There are a load more improvements and tweaks in 1.4.3 that are fully explained in the original announcement including,

Performance tweaks across CSS and Traversal modules
Tweak to allow pausing the DOM Ready event until explicitly specified
Event helpers can now be passed data objects

Now onto jQuery Mobile....

Microsoft's Trio of jQuery Plugins

Fri Oct 15 23:00:00 UTC 2010

Microsoft's 3 jQuery plugins have been accepted as official plugins (with templating being absorbed into the core in jQuery 1.5). Lets have a quick look at what they provide.

There are a ton of blog posts and documentation out there describing what these plugins do,

I am going to ignore Globalisation for now as it takes too long to put a worthwhile demo together and it's alreaqdy heavily explained on the GitHub page linked above. So lets take the other plugins and use them to create "the best todo app evar"

ToDo App

Lets take a simple to do app the you can add Tasks to (along with associated comments) then save them to the server. Simple design like so,

[[posterous-content:iesdxxncdohkDnbunIad]]

HTML

So first things first lets get the HTML out of the way,

<h1>ToDo List</h1>
<form id="new-task-form" action="\">
    <p>
        <label for="task">Task:</label>
        <input type="text" id="task" name="task"/>
    </p>
    <p>
        <label for="note">Note:</label>
        <textarea id="note" name="note" rows="5" cols="30"></textarea>
    </p>
    <input type="button" value="Add" id="add"/>
    <input type="button" value="Save" id="save"/>
</form>
<h2>Current ToDos</h2>
<ul id="todo-list"></ul>

JavaScript

Next up is the interesting part - JavaScript,

    var model = {}, // empty model object
        list = $("#todo-list"),
        form = $('#new-task-form'),
        addBtn = $("#add"),
        saveBtn = $("#save");

    // link ui to model
    form.link(model);

    // handle onclick
    addBtn.click(function () {
        // clone the data model
        var item = $.extend({}, model)
        // generate template
        $.tmpl("<li>${task}</li>", item).appendTo(list);
    });

    // handle save
    saveBtn.click(function () {
        // pull back the data
        var todos = list.find("li").map(function () {
            return $(this).tmplItem().data;
        }).get();

        // save the object
        $.post("/Home/Save", todos, function () {
            alert("Todos Saved");
        });
    });

That's it. So lets break it down and see exactly what is happening.

Variable Declarations

var model = {}, // empty model object

First thing we do is declare a model object. Simply an empty object for our purposes but could be pre-populated with data that can be bound.

Model Linking

Model linking is effectively 2 way data binding between a data context and a form (1 way optional and configurable mappings possible see API).

    // link ui to model
    form.link(model);

We just want simple 2 way binding so that is all we need. This sets up the 2 form inputs so they are bound to our model object so any changes will be reflected accordingly.

Add Functionality

    // clone the data model
    var item = $.extend({}, model)
    $.tmpl("<li>${task}</li>", item).appendTo(list);

First things first we need to take a clone of the current object as this is bound/linked to the form and if we don't clone it we will be referencing the same object over and over. This wont affect the UI but templating has some nice data centric features to pull back our data later and we need this.

Next we create a very simple template passing the cloned model into the template call. Finally we append this new LI to the UL list.

Save Functionality

The interesting part of the save functionality is how we actually pull back the list of data to be sent to the server.

    // pull back the data
    var todos = list.find("li").map(function () {
        return $(this).tmplItem().data;
    }).get();

Templates are data centric. That is, when they are created the original data that was used to generate them is stored in the jQuery data cache and can be retrieved using tmplItem. So rather than having to scrape the data back from UI (where are you going to get the comment value from?) you can simply ask for the elements data context.

So...

That's it. Some very nice data centric (3rd time in as many paragraphs) plugins. Sure you can do this stuff without these plugins but it would certainly not be as elegant. This is only skimming the surface of the data linking and templating plugins, there is a heap of stuff that I didn't cover,

One way binding
Value converters for binding
Programatically change data values
Adding custom linking behaviour
Generate templats from embedded script templates
Complex and nested templating
Statically define a reusable template

Modernizr & RequireJS

Wed Oct 13 23:00:00 UTC 2010

I've been pulling a lot of JavaScript libraries and utilities together to build up a NuPack server as it's by far one of the best ways to distribute client side resources during development. Some of these libraries I have used myself and thought it might be worth mentioning some of them.

Modernizr

Modernizr is a feature detection library that allows us to test the existence of certain HTML5 features within the browser. This is not only useful when constructing a page and applying styling of JS widgets but also for detecting things such as WebWorkers, Web DB, application cache etc. It supports many things,

@font-face, Canvas, Canvas Text, HTML5 Audio, HTML5 Video, rgba(), hsla(), border-image:, border-radius:, box-shadow:, opacity:, Multiple backgrounds, CSS Animations, CSS Columns, CSS Gradients, CSS Reflections, CSS 2D Transforms, CSS 3D Transforms, CSS Transitions, Geolocation API, localStorage, sessionStorage, SVG, SMIL, SVG Clipping, Drag and Drop, hashchange, X-window Messaging, History Management, applicationCache, Web Sockets, Web Workers, Web SQL Database, IndexedDB, Input Types, Input Attributes

In most cases native support for a feature is always better than an alternative CSS hack or JS script. Rounded Corners are a good example of this. A lot of, but not all, browsers already support rounded corners natively but in a world where people think a site should be identical in all browsers it is often necessary to provide your own solution. This means adding extra work for all browsers. This is where Modernizr comes in. Rather than applying non-native rounded corners to browsers that support them nativley we can do this,

<script type="text/javascript">
    if(!Modernizr.borderradius){
        $(".borderradius").roundedcorners();
    }
</script>

That way we can always defer to the browser and remove unnecessary processing. But lets take it step further. Obviously for the majority of browsers pulling down a rounded corner plugin is completely pointless as it will never be used. What can we do about that?

Require.JS

Require.JS is one of many module/script loaders available but the good thing about Require.JS is that is extremely simple. Essentially it injects a script file into the head of the document and fires a callback when ready (there is a bit more configurability held within but that's the gist of it). So lets take the above example and see what we can do about it. We don't want to load scripts unnecessarily (lets leave that up to ASP.NET WebForms, ugh.... ) so lets only load the rounded corner script when needed,

<script type="text/javascript">
    if(!Modernizr.borderradius){
        require(["scripts/jquery-corners.js"], function(){
            $(".borderradius").roundedcorners();
        });        
    }
</script>

So basically we only get the extra script pulled down in the few browsers that don't support rounded corners, less processing, less traffic (as long as require.js file size < total size of optional scripts).

Nice.

Obviously you need to be a bit more strategic about where you place this code as you can get FOUC if placed after document has fully loaded. Another issue is that Require.JS intentionally doesn't support CSS as there is no guaranteed way to ensure it has loaded across all browser. Thats an easy fix though,

function loadCss(url) {
    var link = document.createElement("link");
    link.type = "text/css";
    link.rel = "stylesheet";
    link.href = url;
    document.getElementsByTagName("head")[0].appendChild(link);
}

As always I've put together a nice wee demo app that makes use of these technologies.... coming to a NuPack server near you too.

svn://subversion/Playground/trunk/jameshu/Src/Kainos.Internal.RequireAndModernizr

Let me know if that works as it uses NuPack (but I don't think that is a requirement to open the project).

Outside Events jQuery Plugin

Mon Oct 11 23:00:00 UTC 2010

Outside Event Plugin doesn't have a great deal of uses but it does solve one very important issue in client side code - Diminishing Responsibility. Before I explain how it does this I need to explain what it does.

Outside events are a new set of events provided by the plugin taht are fired on the registered element when the event happens OUTSIDE that element. Imagine a click event

$("#somelement").bind("click", function(){
    // this element was clicked
});

Everytime that #somelement gets clicked the event fires. An outside event is the opposite of that so,

$("#somelement").bind("outsideclick", function(){
    // some other element was clicked
});

This event fires when any other element is clicked that WASN'T #somelement (or whatever query you use - we are not stuck to a single element). Outside events that are supported include (some more useful than others),

clickoutside, dblclickoutside, focusoutside, bluroutside, mousemoveoutside, mousedownoutside, mouseupoutside, mouseoveroutside,mouseoutoutside, keydownoutside, keypressoutside, keyupoutside, changeoutside, selectoutside, submitoutside.

Great, cool but what use is this? I've already said it helps solve the Diminished Responsibility issue but what does that mean?

Diminished Responsibility

Imagine an example where there a popup notification on a web page. It's not obtrusive but should go away when the user clicks off it. How do you do this? The simple answer would be to add a load of (or delegate) click events and hide the element so your handlers would start to look like this [very contrived BTW],

$("#somelement").bind("click", function(){
    if($("#dialog").is(":visible")){
       $("#dialog").hide();
    }
    // perform function
});

Starts to get a bit messy if we have a lot of handlers and is actually bad practise as the popup notification should be as self contained/self managing as possible. Outside events solve this by allowing you to push responsibility back onto the widget/element that needs to "do something". Neat.

The Magic

Funny thing is while this looks like it may be complicated to implement it's really not. Thanks to the power of event delegation and event bubbling all the plugin does is attach the inverse of the outside event (outsideclick becomes click) to the document element and when a click event is fired the handler checks if the clicked element is an element registered with outsideclick and if not fires the event.

Easy peasy though it does open up the possibility of infinite event triggers but that's not the fault of the plugin :-P.

ASP.NET MVC 3 Preview 1 First Look

Tue Jul 27 23:00:00 UTC 2010

[[posterous-content:6xLmqhke9CMHGNqiaUG7]]

A much better read than the ramblings below:
http://weblogs.asp.net/scottgu/archive/2010/07/27/introducing-asp-net-mvc-3-preview-1.aspx

So I had a chance to take ASP.NET MVC 3 Preview 1 out for a test drive today to see what was new and interesting. For a Preview release it seems fairly solid - there is a lot of low level stuff introduced (new Interfaces etc) which is, I assume, going be used to add new features in Preview 2, 3, 4 whatever. There is nothing MASSIVE per say in fact it really feels like the next logical step in the MVC roadmap rather than some crazy reimagining. That's good by the way - backwards compatibility maintained and life goes on as normal. The most obvious "issue" I have with this release is that it's NET4/VS2010 only which means if it came out tomorrow we'd probably not have the choice of making use of it due to the snails pace of the "enterprise". Anyways enough preamble, down to the juicy stuff - what's this release giving us?

Razor

Preview 1 introduces Microsofts new ViewEngine, Razor, to the MVC world (WebMatrix aside). Currently there is no syntax highlighting/intellisense for it but it'll still compile and generate views just fine. I don't want to dive into Razor in this post but it's a weird syntax. It's certainly feels much neater than the WebForms View Engine and for a weird reason it felt quite natural to use it - want to embed some serverside code? Just type @ and you drop into serverside mode but with the advantage of actually still being able to embed HTML content without having to escape it. The render just knows. It's strange but good.

@inherits System.Web.Mvc.WebViewPage<Models.Person>
@{
    View.Title = "Home Page";
    LayoutPage = "~/Views/Shared/_Layout.cshtml";
}

<div>
    @Html.ValidationSummary()
    @{Html.EnableClientValidation();}
    @using(Html.BeginForm()){
        <p>Name:</p>
        @Html.TextBoxFor(p => p.Name)
        <div>@Html.ValidationMessageFor(p => p.Name)</div>
        <input type="submit" value="Submit"/>
    }
</div>

On a side note when you create a new view MVC 3 and VS2010 detects installed view engines so you can EASILY pick and choose which is very nice - though no ViewEngine currently supports this e.g.

[[posterous-content:Ryck0mqEkX92dSBGu1n4]]

Model Validation

There is some nice stuff in this area. In MVC you where restricted to validating properties of a model (by default using data annotations) and if you wanted to validate entire models or perform validation in model scope (e.g. compare 2 fields) you had to write custom code.

MVC3 provides model-level validation in 2 ways,

1. Using Validation Attributes

2. Using the new data annotations interface IValidateObject

Validation attributes are the same attributes that the default validation strategy currently uses except you now have access to the model and ValidationResult has been tweaked to allow broken rules to be bound to multiple fields e.g.

public class PasswordsMustMatchAttribute : ValidationAttribute
{
    protected override ValidationResult IsValid(
        object value, ValidationContext validationContext)
    {
        var model = validationContext.ObjectInstance as Person;
        if (model.Password == model.PasswordConfirm)
        {
            return ValidationResult.Success;
        }

        return new ValidationResult(
            "Password and Password Confirmation must match",
            new string[] { "Password", "PasswordConfirm" });
    }
}

IValidateObject interface is the new way to provide validation of a Model object across all MS technologies (EntityFramework, DynamicData, Silverlight and ADO Data Services) and it behaves much better than the decoupled attribute stuff - at least in the preview.

If model implement the IValidateObject interface they are expected to implement the validate(ValidationContext context) method which returns an enumerable of ValidationResults. The Person model in my sample app gives an example of this,

public class Person : IValidatableObject
{
    public string Name { get; set; }
    public string Nickname { get; set; }
    public string Password { get; set; }
    public string PasswordConfirm { get; set; }
    public int Age { get; set; }

    public IEnumerable<ValidationResult> Validate(
        ValidationContext validationContext)
    {
        if (Name.Equals(Nickname))
        {
            yield return new ValidationResult(
                "Name and Nickname cannot be the same",
                new string[] { "Name", "Nickname" });
        }
    }
}

Global Filters

More of a convenience than anything, global filters allow you to apply the same Action Filters to EVERY controller action within an application. Useful for all tha AOP style stuff like logging etc. Taking the new KDF Action Logger we can log every action invocation by pushing the attribute type into the global filters via global.asax like so,

GlobalFilters.Filters.Add(new LoggingFilter());

Thats it really. Nice and convenient.

Json Model Binding

Typically model bindng is done through creating an new instance of the model and attempting to bind up properties against the names of the values submitted by the client. This doesn't really work in the Ajax world because the body of a POST request is generally a single unnamed parameter which represents a JSON encoded string of the client request data. In MVC2 we had to write custom binders to parse the string and bind to a model but in MVC3 this is provided automatically. Again another nice to have.

Other stuff

· Dynamic ViewModel - Simply a dynamic bag allowing us to use ViewModel.SomeProperty vs ViewData["SomeProperty"]

· New hooks for IoC/Dependency Injection through the lifecycle

· New ways to expose Client Validation and Model Metadata which "should" allow an easier way to create a single point of validation that can be used across both client and server

· New view results - HttpStatusCodeResult and Permanent Redirects

Summing up

MVC3 Preview 1 is a solid enough release but ultimately it seems to be a foundation, albeit a solid one. Given the previous release schedule I expect another release within the next few months. What can we expect for future releases? See here.

CSS Hacks for IE - Unnecessary

Wed Jan 13 00:00:00 UTC 2010

There are lots of CSS tricks floating aroud to accomodate IE quirks and more specifically in most cases IE6 the problem is these hacks - which is exactly what they are - invalidate your CSS and break future-proofing (what if a another browser learns to understand this hack with different rendering effects?). These things are parsed by every browser that comes across them - the browser then has to attempt to parse it, fail and then carry on - not exactly the most performant styling mechanism! They are useful and often quite nice but aren't exactly self explanatory making them a maintenance nightmare if someone doesn;t understand that order can be very important in these cases.

So - Why hack when a system already exists?

Hello Conditional Comments.

Conditional Comments are a feature implemented in IE (though other browsers COULD implement them if they wanted to but none have). It's a specially marked up HTML comment node (so you include it in your HTML not CSS) that is only parsed by IE. A few examples are below....

<!--[if IE 6]>
IE specific Styling
<![endif]-->

<!--[if IE 6]>
IE6 Styling
<![endif]-->

<!--[if gt IE 6]>
IE7 + Styling
<![endif]-->

So basically you have you all normal CSS link tag then you can include a conditional comment to cater for specifics to IE or some specific version or rnge of versions. OK you would still need hacks for the other browsers but NIWater only had to use IE6 conditional comments (no hacks for other browsers) and looks fine in Chrome/Safari/FF - plus it means you CSS is valid and IE specific CSS is only pulled down by IE slightly reducing page weight (very slightly most likely!!!).

.less - Dynamic CSS for .NET

Mon Nov 23 00:00:00 UTC 2009

Link: http://www.dotlesscss.com/

There is a Ruby library (LESS) that gets a fair bit of buzz in Ruby circles. It's purpose is to introduce the DRY principle into CSS but also adds a lot of other useful features. Well now there is a .NET port of it - so what can it do (ripped from the original page as this pretty much sums it up)?

Variables

Variables allow you to specify widely used values in a single place, and then re-use them throughout the style sheet, making global changes as easy as changing one line of code.

@brand_color: #4D926F;

#header {
  color: @brand_color;
}

h2 {
  color: @brand_color;
}

And variables are a god-send in CSS world though other people disagree (http://www.w3.org/People/Bos/CSS-variables) - but at least this is still generating plain old CSS for consumption

Mixins

Mixins allow you to embed all the properties of a class into another class by simply including the class name as one of its properties. It's just like variables, but for whole classes.

.rounded_corners {
  -moz-border-radius: 8px;
  -webkit-border-radius: 8px;
  border-radius: 8px;
}

#header {
  .rounded_corners;
}

#footer {
  .rounded_corners;
}

Operations

Are some elements in your style sheet proportional to other elements? Operations let you add, subtract, divide and multiply property values and colors, giving you the power to do create complex relationships between properties.

@the-border: 1px;
@base-color: #111;

#header {
  color: @base-color * 3;
  border-left: @the-border;
  border-right: @the-border * 2;
}

#footer {
  color: (@base-color + #111) * 1.5;
}

Operations are "OK" i guess but I am not sold on them.

Nesting

Rather than constructing long selector names to specify inheritance, in Less you can simply nest selectors inside other selectors. This makes inheritance clear and style sheets shorter.

#header {
  color: red;
  a {
       font-weight: bold;
       text-decoration: none;
    }
}

I like the nesting as it moves the structure closer to the nexted DOM tree format that you are actually trying to map against

Compression & Caching

Now you can configure if you enable Caching and minifying or the CSS output.

<dotless 
    minifyCss="false" 
    cacheEnabled="true" />

It's still in beta and I haven't had a chance to properly tinker with it but it's a great idea as CSS bloat is something that is often overlooked and can make rebranding a nightmare if not carefully managed. I might post more on this when I get a chance to have proper play with it.

Lazy Function Definition Pattern

Sun Oct 18 23:00:00 UTC 2009

Have a look at this..

function getTemplate(id){
  var scripts = document.getElementsByTagName("script");
  for(script in scripts){
    var sid = script.getAttribute(id);
    if(id==sid){
      return script.innerHtml;
    }
  }
}

Fairly simple function that loads the contents of a specific script tag (based on id) so that it can be used as a template (for something). The reasons why you would do this is not important but the implementation of this function is. Every time you call this function you are re-selecting all the tag elements on the page which is just plain wrong. A quick fix for this would be to cache the results after the initial call and use the cache.

var cache = document.getElementsByTagName("script");
function getTemplate(id){
  for(script in cache){
    var sid = script.getAttribute(id);
    if(id==sid){
      return script.innerHtml;
    }
  }
}

This would allow the the costly getElementsByTagName to be called once and the results reused. This however starts polluting the global namespace with unnecessary variables plus even if the getTemplate function is never used you still have the cost of one call to getElementsByTagName which is pretty poor. You can get around these issues fairly easily by extending the actual function object itself

function getTemplate(id){
  if(!getTemplate.cache){
    getTemplate.cache = document.getElementsByTagName("script");
  } 
  for(script in getTemplate.cache){
    var sid = script.getAttribute(id);
    if(id==sid){
      return script.innerHtml;
    }
  }
}

The benefits of this approach are that you aren't polluting the global namespace and you are only making the call to getElementsByTagName ONLY when it is needed but it does still have an issue. The cache should only be available to the getTemplate functions internal logic but it is currently exposed allowing other functions to remove or update it.

Introducing the Lazy Function Definition Pattern. Thanks to the dynamic nature of JavaScript functions can actually redefine themselves at runtime and this allows us to provide private, lazy, one time processing without polluting the global namespace. Lets see it in action...

function getTemplate(id){
  var cache = document.getElementsByTagName("script");
 
  getTemplate = function(id){
    for(script in cache){
      var sid = script.getAttribute(id);
      if(id==sid){
        return script.innerHtml;
      }
    }    
  }
 
  return getTemplate(id);
}

The big change here is the middle statement. getTemplate is actually redefining itself. So what happens here?

On the first call getElementsByTagName is called and stored in the cache variable
The function then redefines itself as simple function that simply loops through the already defined cache
The function then calls itself (the new self!)

Thanks to the creation of a closure the redefined function actually has access to the variables of the old definition - cache. This means that cache is private to the new function, only called once and not part of the global namespace. Exactly what I wanted. Programatically as well it's a better solution than the ones above because there are no checks to see if cache is defined saving a few cycles each call.

JavaScript Workshop

Thu Oct 01 00:00:00 UTC 2009

LINQPad: Don't let the name fool you

Tue Aug 25 23:00:00 UTC 2009

LINQPad Website

I've been playing with this little app for a few days now and it's exactly what I was looking for. For the last few years I was knee deep in the crazy world of Java and in my never ending attempt to be different I researched Groovy quite a bit. One of the things that came with the installation of Groovy was a console that let me write off the cuff Groovy or Java statements without having to fire up an editor and stick all that public static void main(string[] args) boilerplate around my code, compile it and run it. I used this console a lot when debugging little bits of code, testing theories, seeing a dump of an object etc and it sped up my development no end. Since getting involved in the .NET world any time I've wanted to write small test apps I've had to either create a new Unit Test or new Console Application and it's a lot slower to do achieve the same results.

This is where LINQPad comes in. LINQPad lets you write C# (and VB) statements, expressions, programs etc and evaluates them as they are without having to write a lot of boilerplate. If your code returns an Object it can be dumped in a very neat fashion into the output window along with the generated IL and some other stuff. LINQPad, as the name suggests also lets you hook up to SQL Server, the Entity Framework, Attachable Databases etc and perform LINQ Queries, showing you the generated SQL as well.

LINQPad reports the execution time in the status bar, so you won't have to manually create a Stopwatch class for performance testing.
Want to test a variation of your snippet? Ctrl+Shift+C instantly clones your snippet so you can run another version side-by-side.
You can return to saved queries in single click, thanks to the My Queries treeview. Some people are using LINQPad as a scripting tool!

The licensed version gives you full Intellisense for your code plus 2 books worth (C# in a Nutshell and LINQ in Action) of working samples. Anyways here are a few screenshots of it in action. Like I said the name nearly made me skip over it but luckily I decided to try it and found what I was after!

[[posterous-content:HEtroelnwcgbGuFkaJnG]]

JavaScript Method Overloading

Mon May 18 23:00:00 UTC 2009

I was recently asked a question if method overloading in JavaScript was good practise, or even possible, so I thought I'd spread share my thoughts on this to anyone who bothers to read my ramblings. JavaScript doesn't offer method overloading directly but it does offer features that can be used to facilitate overloading. Imagine these C# method signatures,

// Joins 2 strings
public string Join(string a, string b)
{
    return a+b;
}
 
// Joins an array of strings
public string Join(string[] a)
{
    return a.join('');
}
 
Join(new string[2] {'a', 'b'}); // <-- 'ab'
Join('a','b'); // <-- 'ab'

So depending on the arguments passed to the function the correct method will be called. Now this style of overloading isn't possible in JavaScript.

// Joins 2 strings
function join(a, b){
    return a+b;
}
 
// Joins an array of strings
public join(a){
    return a.join('');
}
 
join(['a','b']) // <-- 'ab'
join('a','b') // <-- ERROR! (join is not a function of string)

As I have said JavaScript has no notion of method overloading so all that happens here is that the join function get redefined with the last occurrence of the function definition. However thanks to JavaScript's typeless nature and optional arguments overloading can be achieved (or at least simulated depending on your viewpoint).

function join(a, b){
    if(typeof(a) == 'string'){
        return a+b
    }else{
        return a.join('');
    }
}
 
join(['a','b']) // <-- 'ab'
join('a','b') // <-- 'ab'

Fixed! OK so that may seem simple enough and in this case it is but there are 2 things you need to consider before actually putting this into practise.

Signature Detection

Obviously the first thing you need to do when overloading is detect the intention of the user - what method signature are they using. This is the biggest issue you'll probably come across, especially when dealing with arguments of typefunction or array. These posts explain the various issues with type detection and functions/arrays but to summarise

typeof returns "object" for Arrays
An array created in a frame/iframe does not share the same prototype as another Array in the parent document or another frame/iframe which make instanceof, typeof and [].constructor fail
Attempting to detect features such as methods (splice, join) and properties (length) can fail if the a non-array has matching methods/properties
Certain browsers report objects as functions making typeof == function fail
Host methods (browsers provided methods) sometimes do not report themselves as functions

The list goes on! So you need to be aware of the target browsers quirks and work with them. Another way to detect signature is to use the arguments psuedo-array but this is only useful in situations where your overloaded methods accepts a varying number of arguments e.g.

function join(a, b){
    if(arguments.length == 2){
        return a+b
    }else{
        return a.join('');
    }
}

But then this is open to abuse by ignorant/stupid/malicious coders as it doesn't really imply how the function should behave unless commented well (yeah right!!).

Code Bloat

You need to be sure you aren't abusing this overloading strategy as it can lead to spaghetti code. Functions that attempt to do many things with similar signatures result in loads of argument testing and lot's of indented code. jQuery code is riddled with this due to the way almost every method acts as either a setter, bulk setter or getter method and it's run into bugs in the past due to this. Also the jQuery (or $) function is overloaded to do 5 different things so looking at the init function you'll see that it is a big if..elseif..elseif..elseif..else block which is pretty gruesome and breaks extensability and modularity of the code base.

Conclusion

Overloading is possible in JavaScript but needs to be used with caution. Not only can it lead to very hard to find bugs due to type detection in different browsers but it can also make code harder to understand and extend, not to mention that amount of boilerplate code you need to write to detect the correct method signature etc.

UpdatePanels & Alternatives

Tue Apr 14 23:00:00 UTC 2009

I'm not sold on ASP.NET UpdatePanels. Conceptually they are quite nice but their implementation is nasty. So much so I'd be happy to say the should be considered an anti-pattern. My reasons? Well...

They generate too much JavaScript
It's too "development focused" - it puts saving time ahead of a clean performant solution
Ajax calls should be data focused (returning XML, JSON etc), UpdatePanels are content focused
Generates a massive amount of network traffic in comparison to alternative solutions (including ViewState and markup)
UpdatePanels use innerHTML to replace contents which orphans bound events (leakage) that need to be rebound each time (extra work).

But what are the alternatives? Page Methods and Scripted Services are a good starting point. The good things about these alternatives is that they are lightweight - they send JSON data to and from the server so your network traffic is greatly reduced. Here is a simple example of using a page method - I'll not cover Scripted Services as they are very similar and you can find info about them on the net.

In your code behind you can add a [WebMethod] attributed method

using System;
using System.Web.Services;
 
namespace WebformControls{
    public partial class PageMethods : System.Web.UI.Page{        
        [WebMethod]
        public static string GetCurrentDateTime(){
            return DateTime.Now.ToString();
        }
    }
}

And turn on EnablePageMethods attribute of the pages ScriptManager

<html xmlns="http://www.w3.org/1999/xhtml" >
    <head runat="server">
        <title></title>
    </head>
    <body>
        <form id="form1" runat="server">
            <asp:ScriptManager runat="server" EnablePageMethods="true"></asp:ScriptManager>
        </form>
    </body>
</html>

What happens next is that when the page is rendered a JavaScript Proxy Class will be generated called PageMethods that contains a function call GetCurrentDateAndTime. It'll look something like this.

var PageMethods = function() {
    PageMethods.initializeBase(this);
    this._timeout = 0;
    this._userContext = null;
    this._succeeded = null;
    this._failed = null;
}
PageMethods.prototype = {
    _get_path: function() {
        var p = this.get_path();
        if (p) return p;
        else return PageMethods._staticInstance.get_path();
    },
    GetCurrentDateTime: function(succeededCallback, failedCallback, userContext) {
        /// 
        /// 
        /// 
        return this._invoke(this._get_path(), 'GetCurrentDateTime', false, {}, succeededCallback, failedCallback, userContext);
    } 
}
PageMethods.set_path("/PageMethods.aspx");
PageMethods.GetCurrentDateTime = function(onSuccess, onFailed, userContext) {
    /// 
    /// 
    /// 
    PageMethods._staticInstance.GetCurrentDateTime(onSuccess, onFailed, userContext);
}

Now all we have to do is use it. This is easy...

<html xmlns="http://www.w3.org/1999/xhtml" >
    <head runat="server">
        <title></title>
        <script type="text/javascript">
            function getDate() {
                PageMethods.GetCurrentDateTime(function(data) {
                    document.getElementById("results").innerHTML = data;
                })
            }
        </script>
    </head>
    <body>
        <form id="form1" runat="server">
            <asp:ScriptManager runat="server" EnablePageMethods="true"></asp:ScriptManager>
            <a href="#" onclick="getDate()">Get Date!</a>
            <div id="results"></div>
        </form>
    </body>
</html>

That wasn't too difficult now was it? But we aren't done yet. Sure this solution works but there is still one problem - We are including the ScriptManager in our page which means event though all we are doing is getting a date back from the server we are still required to pull down at least 160k (in it's most compressed & gzipped form) of framework 90% of which we aren't even going to use.

Calling PageMethods and ScriptedServices using jQuery

Obviously a request from client to server is always done through HTTP it doesn't actually matter how it is handled on the client. So it's easy to swap out the ASP.NET Ajax framework for jQuery. You'll probably be asking why? Well the long and short of it is that regardless of what the requirements say you (or someone else on the project) WILL end up writing Javascript and jQuery is the better choice...

It's a better, smaller, compact and faster framework
It's much better documented and easier to learn
It's extensible via plugins so you don't need the whole "kitchen sink" to do simple things
It can do everything ASP.NET Ajax can do
It's technology agnostic so you can jump to a Java project and use the same solutions

I could go on, honestly I could, but that's not what this post is about. There are a few nuances in calling PageMethods and ScriptedServices, namely

The request content-type must be application/json; charset=utf-8
The request must be a POST request
Data sent to the server must be encoded as a JSON string or "{}" if empty
Standard ASP.NET JSON serialization doesn't convert DateTime to a proper JSON date
The JSON data in the request must map directly to arguments of the function

Failure to comply with any of the above will result in either a security error or, usually, the entire page being rendered back to the response. So calling the above PageMethod using jQuery is as simple as this....

$.ajax({
  type: "POST",
  url: "PageMethods.aspx/GetCurrentDateTime",
  data: "{}",
  contentType: "application/json; charset=utf-8",
  dataType: "json",
  success: function(msg) {
    $('#result').html(msg)
  }
});

Not bad, but there is a lot of boiler plate code there. Thankfully it's easy to write a jQuery plugin to abstract out most of this and make the calls simple. So all in all even with the 2 plugins (JSON and .Net Services) and the jQuery framework we are only forced to pull down 60k of scripts (minified but NOT gzipped) so you could see this reduced to 25-30k all in with gzipping turned on.

Yo! Briefcase: index

Bashing Ambiguity

How to do unnecessary things in long winded ways...

Quick and Dirty JSON Builder for Java

ansible-console: An Interactive REPL for Ansible

Tabularizing output of shell scripts with column

Enter column

Functional Collections in Scala (Lightning Talk)

Kotlin - 2 Years On

alias, union and case in Crystal

alias

union

case

entr

Install it

Use it

Further betterness

An example of refactoring Scala code

Set the scene

The current solution

Why not just use a var?

Why do we even need the var?

Improving the flow

Using Option as a pipeline

Compose a pipeline

So which is best?

Refactoring Pattern Matching in Scala

Setting the stage

Decisions using pattern matching

Deeper matching

Option.collect

Summary

On Popularity

Birth of a Star

Death of a Star

Starry Starry Night

Stop look and listen. Oh and think as well. Do lots of that.

The Twelve Factor App (Redux)

More Betterness

More Betterness

Refactoring Towards Transducers

Towards Transducers

Deconstructing Your First Elm App

Deconstruction

Using start-app

TO THE CODE!!

Starting Point

Model

View

Actions

Update

Running the App

Wrapping Up start-app

Removing start-app

The Mailbox

Reactive Model

Mapping Main

And rest...

Zombie Dice Score Card in Elm

Types

Error Messages

HTML

It's just one big fold

Conclusion

window.prompt() in Elm

Basic Implementation

Notes

Zombie Dice Score Card

Zombie Dice Score Card with Reagent

Tenzing

Boot

Reagent

Source

Plans

Footnote

Meteor: An Introduction by an Expert Beginner

Isolation

Road Trippin'

GPS

Roadworks and accidents

Enter `column`

`alias`

`union`

`case`

Why not just use a `var`?

Why do we even need the `var`?

Using `Option` as a pipeline