Your seat in the clouds awaits you

Ce billet de blog est également disponible en français ici.

Back in November 2011 easyJet announced that starting in the spring of 2012 we would begin a trial of allocated seating and on April 12th we went live on five routes from London Luton and Glasgow. Since then we have gradually extended this trial and we are now offering allocated seating on almost all routes on our network and by the end of November we will be at 100% operational delivery. This is a huge change for easyJet. Free seating, referred to by many of our passengers as “the scrum”, was part of our DNA. It was how we had always operated. It had become part of the definition of easyJet. As our CEO Carolyn McCall said in the article above, the trial could only be deemed successful if it met all three of the following criteria:

1. It had to increase customer satisfaction. We work hard to have happy customers. It’s another thing that’s part of our DNA. Allocated seating had to really make a difference to the passenger experience. Many people said that they wanted it but, once we gave it to them, would it really make the difference they thought?

2. It had to work operationally. easyJet operates one of the quickest turn-around times in the industry. If boarding passengers into allocated seats was seen to have a negative effect on our On Time Performance (OTP) it would not have been considered viable.

3. It had to work commercially. Allocated seating had to prove itself a commercial success as a revenue generating product.

Trial by fire

This highlights the fact that such a move was a calculated risk. We were not sure it would work but it required significant change and investment to find out. One of the major changes was to our reservation system. Our home-grown reservation system did not support allocated seating. The primary advantage of maintaining a bespoke system is that it can be tailored to your exact business needs. No extraneous functionality cluttering up the works. It does, on the other hand, support bookings from around 58 million passengers a year and take over £4 billion in revenue.

Changing the beating heart of our enterprise, our various sales channels like easyJet.com and our operational systems to support allocated seating was no small undertaking, quite apart from the changes to our operational processes. Making those changes to support a trial, an experiment? That called for a quite special approach.

Our first decision was that we definitely did not want to have to conduct open-heart surgery on our reservation system to add this functionality. The I/O load from selling 58 million non-specific seats a year is already a veritable fire-hose. Scaling and refactoring to support the tracking and locking of over 58 million specific seats on a system that can book up to 1500 seats a minute would be a huge project.

“Seat-allocation-as-a-service”

Would it be possible, we wondered, to buy “seat-allocation-as-a-service” (SaaaS?) from a third party? Get someone else to do the heavy lifting of tracking the availability of every single seat we have on sale while we just stored the output, a few bytes that represented the selections made for the seats we have actually sold?

Apparently not. However the idea of a separate “seat-allocation-as-a-service” solution attached via a very light-weight integration was too attractive to let go so we decided to build our own.

What this means, in summary, is that the tens of millions of seats we have available at any given time are tracked via partitioned SQL Azure databases and cached in the Azure AppFabric Cache. All the logic, business rules and data relating to …

  • selecting seats
  • handling contention for seats
  • aircraft types
  • seating layouts and configurations
  • price bands
  • which passengers can sit where
  • seating access for passengers with restricted mobility
  • algorithms for automatically allocating seats to passengers who chose not to make a selection
  • and the million-and-one other things that have to be taken into consideration when seating an aircraft

…all this is done in the cloud. Even the interactive UI that displays the graphical map of the aircraft is served from Azure and injected into the booking pages on easyJet.com.

seating

The ingenious work to achieve this using JSONP, Ajax and Knockout.js (amongst other things) is a tribute to the fantastic development team at easyJet and may be the subject of a subsequent blog post.

The overall approach however has allowed us to implement an incredibly significant change to the way we operate and sell our flights and deliver it at massive scale without needing to implement much more than small refactorings in our core operational and retail systems. The low cost and massive scale of Azure has made the whole notion of experimenting with something so fundamental an achievable reality. This calculated risk has become a bet we can much more easily afford to make.

Most importantly it has massively reduced the cost of failure. We had to conduct a thorough trial. We couldn’t be sure that it would work. Whether it worked or not was primarily a business decision rather than a technical one.

Now that it has been successful we have delivered a solution that works technically, works operationally, works commercially, improves customer experience and transformed our enterprise. However, if it had not worked and we had needed to turn it all off and walk away, we could have done so without having incurred huge risk, technical debt or cost.

Implementing a REST-ful service using OpenRasta and OAuth WRAP with Windows Azure AppFabric ACS.

I’ve been building prototypes again and I wanted to build a service that exposed a fairly simple, resource-based data API with granular security, i.e. some of my users would be allowed to access one resource but not another or they might be allowed to read a resource but not create or update them.

To do this I’ve used OpenRasta and created an security model based on OAuth/WRAP claims issued by the Windows Azure AppFabric Access Control Service (ACS).

The client can now make a rest call to the ACS passing an identity and secret key. In return they will be issued with a set of claims. A typical claim encompasses a resource in my REST service and the action(s) the user is allowed to perform, so their claim set might show that they were allowed to execute GET against resource foo but not POST or PUT.

In my handler in OpenRasta I add method attributes that indicate what claims are required to invoke that method, for instance in my handler for resources of type foo I might have the following method  :

[RequiresClaims("com.somedomain.api.resourceaction", "GetFoo")]
public OperationResult GetFooByID(int id)
{
	//elided
}

In my solution I have created an OpenRasta interceptor which examines inbound requests, validates the claim set and then compares the claims required by the method attribute to the claims in the claim set. Only if there is a match can the request be processed.

After a few tweets with @serialseb I refactored the above into an IAuthenticationScheme that validates the claims, leaving the original OperationInterceptor to check the claims required by the method to be invoked. I also added an extension method so that the whole thing can be fluently configured like so :

ResourceSpace.Uses.AzureClaimsAuthenticationScheme();

I was going to write a long blog post about how to build this from scratch with diagrams and screen shots and code samples but I found that I couldn’t be arsed. If you’d like more info more on how to do this just drop me a line. In the meantime I’ve dropped the source files as follows :

Azure AppFabric ACS Gotchas : Longest Prefix Matching

I recently got bitten by a bit of Access Control Service logic related to the way it identifies which scope to issue claims for.

I have a service namespace foo. My Azure Service Bus scope for this namespace is therefore http://foo.servicebus.windows.net. When I create the solution for this namespace two ACS instances are created. One is https://foo.accesscontrol.windows.net which is a general ACS and the other is https://foo-sb.accesscontrol.windows.net which is scoped specifically to the Service Bus. This second ACS has a default Token Policy and Scope which I cannot change. NB : this has now changed. When you provision an AppFabric namespace now you can sepecify which services should be available (ACS, Service Bus and Cache). If you specify Service Bus you will get the bus-scoped ACS instance. You only get the generic ACS if you specifically request ACS as a service.

In my Service Bus solution I am exposing endpoints at http://foo.servicebus.windows.net/bar and http://foo.servicebus.windows.net/baz. I have an issuer (Alice) with claims for the scope http://foo.servicebus.windows.net who is able to create endpoints at both http://foo.servicebus.windows.net/bar and http://foo.servicebus.windows.net/baz. I have another issuer (Bob) who is able to send messages to endpoints at both http://foo.servicebus.windows.net/bar and http://foo.servicebus.windows.net/baz.

I introduce a new issuer (Ivan) to whom I only want to grant access to http://foo.servicebus.windows.net/baz. To this end I create a new scope specifically for http://foo.servicebus.windows.net/baz and create claims for Ivan in this new scope.

Here’s where I get bitten. Alice can still expose endpoints at http://foo.servicebus.windows.net/bar and Bob can still send messages to them. Ivan can send messages to http://foo.servicebus.windows.net/baz but not to http://foo.servicebus.windows.net/bar which is exactly as intended. However, Alice can no longer expose endpoints at http://foo.servicebus.windows.net/baz and Bob could not send to them even if she could. The reason for this is that although Alice and Bob have claims for http://foo.servicebus.windows.net when they try to access anything at http://foo.servicebus.windows.net/baz they automatically fall into the new scope, for which they have no claims. The ACS matches scopes using the longest possible prefix and if there are no claims it will not check parent scopes.

The solution is simple – add new claims for Alice and Bob in the new scope, but the problem is, at first, counter-intuitive.

SQL Azure

Finally getting round to having a play with SQL Azure. You can’t administer it with the usual UI tools like SQL Server Management Studio unless you want to use this slightly kludgy workaround, only SQLCMD 😦 Fortunately,  Julien Hanssens and Martin Balliauw have created SQL Azure Manager which you can install and run instantly using Click-Once. Nice 🙂

More Azure Videos

Back in July, in the events surrounding the Azure announcements at WPC, I was involved in a press event at Microsoft UK where various Microsoft partners and customers talked about their work on the Azure platform. Dave Gristwood and Eric Nelson made some great videos of the event which are available here.

Service (Bus) Browser

I’ve put together a little tool to help with some development work I am doing. While Azure and .NET Services are still in CTP the tooling is a bit limited so, what with necessity being the mother of invention and all that, I’ve spent an afternoon hacking together my Service(Bus)Browser.

Basically, all the endpoints you create in a .NET Services solution – routers, queues, relay endpoints etc – are all listed in the service registry which is exposed as an Atom feed at the base URL of your solution. So, if you create a solution called “MySolution”, your registry root is at https://mysolution.servicesbus.windows.net. Point your favourite web browser at it and you’ll see what I mean.

However, only endpoints whose discoverability policy is set to public will appear unless you also submit the correct credentials when requesting the Atom feed. Here’s an example :

<feed xmlns="http://www.w3.org/2005/Atom">
        <title type="text">Publicly Listed Services</title>
        <subtitle type="text">This is the list of publicly-listed services currently available</subtitle>
        <id>uuid:6bbbd412-81e8-4f30-8629-909d8175e05a;id=558</id>
        <updated>2009-08-05T21:08:29Z</updated>
        <generator>Microsoft .NET Services - Service Bus</generator>
        <entry>
            <id>uuid:f661ac5a-e633-46b7-9d2a-845676c92c82;id=43</id>
            <title type="text">rest</title>
            <updated>2009-08-07T23:06:20Z</updated>
            <link rel="alternate" href="https://mysolution.servicebus.windows.net/rest/"/>
            <link rel="self" href="https://mysolution.servicebus.windows.net/rest/"/>
        </entry>
        <entry>
            <id>uuid:6bbbd412-81e8-4f30-8629-909d8175e05a;id=559</id>
            <title type="text">testqueues</title>
            <updated>2009-08-05T21:08:29Z</updated>
            <link rel="alternate" href="https://mysolution.servicebus.windows.net/testqueues/" />
            <link rel="self" href="https://mysolution.servicebus.windows.net/testqueues/" />
        </entry>
        <entry>
            <id>uuid:6bbbd412-81e8-4f30-8629-909d8175e05a;id=560</id>
            <title type="text">testrouters</title>
            <updated>2009-08-05T21:08:29Z</updated>
            <link rel="alternate" href="https://mysolution.servicebus.windows.net/testrouters/" />
            <link rel="self" href="https://mysolution.servicebus.windows.net/testrouters/" />
        </entry>
    </feed>

We can see three entries in the feed each denoting a node in the registry. Submitting a similar request to the URL of a node will retrieve the Atom feed of its child nodes and so on. Here’s the Atom feed for the testrouters node :

<feed xmlns="http://www.w3.org/2005/Atom">
    <title type="text">Publicly Listed Services</title>
    <subtitle type="text">This is the list of publicly-listed services currently available</subtitle>
    <id>uuid:11e6ffbf-1463-46e5-9a3a-61b7f0966af0;id=349</id>
    <updated>2009-08-08T08:31:00Z</updated>
    <generator>Microsoft® .NET Services - Service Bus</generator>
    <entry>
        <id>uuid:11e6ffbf-1463-46e5-9a3a-61b7f0966af0;id=350</id>
        <title type="text">5922fd42-6120-44cd-b07a-c3a36bd70168</title>
        <updated>2009-08-08T08:31:00Z</updated>
        <link rel="alternate" href="https://mysolution.servicebus.windows.net/TestRouters/5922fd42-6120-44cd-b07a-c3a36bd70168" />
        <link rel="self" href="https://mysolution.servicebus.windows.net/TestRouters/5922fd42-6120-44cd-b07a-c3a36bd70168!(router)" />
        <link rel="subscriptions" href="https://mysolution.servicebus.windows.net/TestRouters/5922fd42-6120-44cd-b07a-c3a36bd70168!(router/subscriptions)" />
        <RouterPolicy xmlns="http://schemas.microsoft.com/netservices/2009/05/servicebus/connect" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
            <Authorization>Required</Authorization>
            <Discoverability>Managers</Discoverability>
            <ExpirationInstant>2009-08-08T08:50:14.9579043Z</ExpirationInstant>
            <TransportProtection>AllPaths</TransportProtection>
            <MaxMessageSize>61440</MaxMessageSize>
            <BufferTimeout>PT10S</BufferTimeout>
            <MaxBufferLength>0</MaxBufferLength>
            <MaxBufferCapacity>0</MaxBufferCapacity>
            <MaxSubscribers>50</MaxSubscribers>
            <MessageDistribution>AllSubscribers</MessageDistribution>
            <PushDeliveryRetries>3</PushDeliveryRetries>
            <PushDeliveryTimeout>PT30S</PushDeliveryTimeout>
        </RouterPolicy>
    </entry>
</feed>

That’s the entry for a router along with an Atom extension element describing the router policy.

Here’s the UI. It’s basically a TreeView for the registry nodes, a PropertyGrid for the node properties and a RichTextBox for the raw Atom.

Service Browser UI

Currently it will manage multiple solutions, walk the Atom tree, display the nodes and their properties, policies and subscriptions. It also allows you to delete queues and routers. Next steps, when I get a bit more time, are to fix the subscription display for the router nodes and to allow the creation of routers and queues (and perhaps subscriptions). Current impediments, other than time, are that the Atom feed to the subscription URL on a router always seems to come back empty even when there are subscriptions 😦

You can download the source here. Feedback and comments welcome unless they are a 17 page treatise on why I should have used [insert your favourite UI stack here] and the [insert your favourite pattern here] pattern 🙂

Please note : Since I wrote this article and posted the code there have been a number of changes to .NET Services, the Service Bus and the Access Control Service, most notably in the November CTP where, amongst other things, we lost Routers and Queues. As a result this code will no longer work. I am working on an updated solution that demonstrates the same functionality which I will post as soon as it’s ready.

Azure Service Bus Presentation and Demo Code

I presented to the UK Azure User Group at Microsoft in London last night. Thanks to everyone who came along and thanks also for all the kind words afterwards. I’m glad so many of you enjoyed it. As promised my slide deck and demo code are now available for download here.

I’ve tried to make the demo as self-contained as possible so anyone can get it up and running easily. However, please read the setup and prerequisite instructions included in the solution and if you have any problems don’t hesitate to drop me an email and let me know.

Also, if you think the .NET Services and Service Bus components of the Azure Services Platform are cool, exciting, compelling, interesting, useful, revolutionary or any of the other terms I heard used last night, please make some noise about it and about anything you are doing with them. Microsoft do monitor the community chatter during a CTP to guage the response to different features and we should be as quick, as loud and as forthright coming forward with our positive responses to these things as we are when they go wrong or we find bugs.

Thanks again.

Click Here to View The Video Titled: UK Azure User Group Meeting#2 Talk #2: EasyJet and Azure (Bert Craven)

Please note : Since this presentation there have been a number of changes to .NET Services, the Service Bus and the Access Control Service, most notably in the November CTP where, amongst other things, we lost Routers and Queues. As a result this code will no longer work.It would also seem that both the UKAzureNet site and ExposareRoom.com have shut down so there’s no video anymore 😦 I will attempt to find a backup 🙂

A Couple of Bugs (or not)

A couple of niggles have come to light today. The first one is that Windows Azure does not seem to support the latest version of the Azure .NET Services Bus. The recent release of the .NET Services SDK, the July CTP, included a new version of Microsoft.ServiceBus.dll (0.16.0.0). This version is not available on Windows Azure. After I had updated my SDK I rebuilt and redeployed a Windows Azure cloud application with a Web Role that calls service bus endpoints. This started throwing all kinds of odd exceptions which was confusing as it (of course) worked fine on my machine. I eventually tracked the issue down and the only way round it at the moment is to set the Copy Local property in the properties of the reference to Microsoft.ServiceBus.dll to true. That way it will get deployed with the application.

The second issue was a kind of two in one. I’m developing some prototypes for Windows Mobile using the .NET Compact Framework 3.5. One of the things I’m working on is a method of keeping mobile devices synchronised via .Net Service Bus queues and routers. There is no native support for the Service Bus on .NETCF but fortunately I only need to make REST calls using a meteor pattern to poll a queue for sync messages.

I had two problems. The first was that I could only poll a queue twice using HttpWebRequest before I’d start getting timeouts. This turned out to be because I was hitting the max connections limit (2 by default), even though I was carefully disposing of responses and closing response streams etc. The second was that I would always get an ObjectDisposedException at System.Threading.WaitHandle.CheckResultInternal when I closed the application. The polling of the queue was of course being done on a background thread, so looking at the two together it did seem that some resource was not being correctly released somewhere.

Looking into the stack trace of the ObjectDisposedException exception I could see references to HttpWebRequest and stream writes. It occurred to me that because I was using REST I was only ever sending headers in my HttpWebRequests, and no body. When I was calling WebRequest.Create() it was possible that the request stream was being opened and because I wasn’t using it it never got closed. As these requests were being created on my background thread they would not dispose correctly and they would continue to consume connection resources and possibly casue thread termination exceptions.

The answer was to close the request stream before sending the request :

try{    HttpWebRequest dequeueRequest = (HttpWebRequest)WebRequest.Create(_QueueHeadUri.AbsoluteUri + "?encoding=asreply&maxmessages=1&timeout=15");    dequeueRequest.ConnectionGroupName = "queueclient";    dequeueRequest.Method = "DELETE";    dequeueRequest.Timeout = 30000;    dequeueRequest.ContentLength = 0;    dequeueRequest.Headers.Add("X-MS-Identity-Token", this._AccessToken);    // this next line is the key     dequeueRequest.GetRequestStream().Close();    using (HttpWebResponse response = (HttpWebResponse)dequeueRequest.GetResponse())    {        if (response.StatusCode == HttpStatusCode.OK)        {            string viaString = response.Headers["X-REQUEST-HTTP-URI"];            Uri viaUri = new Uri(viaString);            if (this.OnMessageDequeued != null)            {                this.OnMessageDequeued(this, viaUri.Query);            }            continue;        }        if (response.StatusCode == HttpStatusCode.NoContent)        {            continue;        }        if (this.OnMessageDequeueError != null)        {            this.OnMessageDequeueError(this, new Exception(string.Format("HTTP Status Code : {0} : {1}", ((int)response.StatusCode), response.StatusCode)));        }    }}catch (Exception ex){    if (this.OnMessageDequeueError != null)    {        this.OnMessageDequeueError(this, ex);    }}

HALO – Part 1

I was preparing a blog post on easyJet’s new HALO patform and the work we’ve been doing with the Microsoft Azure Services Platform and mobile devices. However it seems Computer Weekly have jumped the gun. More to follow, but here’s the first scoop :

Followed By :

For more background to Halo and our adventures in Azure see this blog post with video.

UKAzureNet

The next UKAzureNet meeting will be on Wednesday 29th July at Microsoft’s Cardinal Place building in London. I will be presenting on Azure and the .Net Service Bus in a presentation entitled “Orange meets Azure : If Azure is the answer, what question is easyJet asking ?” The event is free, refreshments will be provided and you can sign up here.
%d bloggers like this: