The Rule of 8 : High Performance Scalable Systems

Wille Faler has an interesting article on his blog where he highlights 8 core principles for building high performance, scalable systems. It’s an extension to the list of Best Practices for Speeding Up Your Web Site, but Wille is focussing on general principles for any scalable system rather than just websites.

In short, we have :

  • Offload the database – Avoid hitting the database, and avoid opening transactions or connections unless you absolutely need to use them.
  • What a difference a cache makes – For read heavy applications caching is the easiest way offload the database.
  • Cache as coarse-grained objects as possible – Coarse-grained objects save CPU and time by requiring fewer reads to assemble objects.
  • Don’t store transient state permanently – Is it really necessary to store your transient data in the database?
  • Location, Location – put things close to where they are supposed to be delivered.
  • Constrain concurrent access to limited resource – it’s quicker to let a single thread do work and finish rather than flooding finite resources with 200 client threads.
  • Staged, asynchronous processing – separate a process using asynchronicity into separate steps mediated by queues and executed by a limited number of workers in each step.
  • Minimize network chatter – Avoid remote communication if you can as it’s slower and less reliable than local computation.

Twitter Scaling with Scala : At Some Point the Cost of Servers Outweighs the Cost of Programmers

There’s a good article here about Twitter’s seemingly controversial move from a pure Ruby stack to Ruby on Rails plus JVM/Scala. What’s most interesting though is why this is seen as remotely controversial. It really illustrates how many people are still slaves to completely irrational and fanatical devotions not only to particular technologies, but pure implementations thereof. There are people leaping in left, right and centre almost falling over each other to demonstrate their unique combination of ignorance and bias. As Todd Hoff points out,

“Twitter didn’t take this large leap out of ignorance or incompetence.”

And Twitter’s Steve Jenson adds, :

“We spent several weeks going over our options, running extensive load tests, and presented our findings to the team at each stage. We did our due diligence.”

But it doesn’t matter how thorough the research and testing or how well founded the argument. The problem is that the fanatics have previously adopted twitter as their mascot or standard bearer and they became its champions. It became the thing they pointed to or waved aloft when arguing the case for the supremacy of their of little pet technology. As a result they made twitter their own, they made an emotional investment in it. Now, when it’s being changed they feel let down, betrayed even. Ridiculous.

How not to scale

Speaking of scale there’s an interesting article here by Bart Smaalders of Sun Microsystems about performance anti-patterns. He has identified a number of these anti-patterns :

  • Fixing Performance at the End of the Project
  • Measuring and Comparing the Wrong Things
  • Algorithmic Antipathy
  • Reusing Software
  • Iterating Because That’s What Computers Do Well
  • Premature Optimization
  • Focusing on What You Can See Rather Than on the Problem
  • Software Layering
  • Excessive Numbers of Threads
  • Asymmetric Hardware Utilization
  • Not Optimizing for the Common Case
  • Needless Swapping of Cache Lines Between CPUs

Although it does appear to have been written from the perspective of “performance over all” and some sections (e.g. the one about layering) ignore that fact that the history of software development has been a gradual trade-off between performance and other equally important things like flexibility, maintainability, portability, modularity, productivity etc, it’s a short, well written article that makes interesting reading.

Really BIG numbers of people

A lot seems to be made in the media about user reactions to changes in services like Facebook. There’s a certain amount of spiteful glee with which these stories are reported. Facebook is being “forced to backpedal”, or “driven into a u-turn” by “user revolt” over changes in its UI/homepage. The media peddles this perception that Facebook just can’t seem to get it right. That every time they make any change there’s this huge uprising and backlash of user opinion against them.

However, last time I checked Facebook had over 180 million users. If it was a country it would be the 6th largest country in the world, between Brazil and Pakistan. So, let’s face it, if Facebook were simply to change the colour of their page header to a slightly lighter blue then just statistically they would be likely to piss off around 20 million people and confuse another 30 million. Firstly, people don’t like change. Secondly, everyone is either an expert about what Facebook should be like or a critic of how it is. Finally, as Mr Gladwell has explained, you can’t trust people’s opinions about what they like or what they want anyway. The vast majority of the these millions of users didn’t even know they wanted Facebook until Zuckerberg and Co. presented it to them. Every time there’s a change at Facebook a dozen or more groups will be formed called things like “Give us back our old Facebook!”. These groups will generally contain all the people who joined the same groups last time there was a change. So, which is it people ? Do you want your old Facebook back ? Or your old, old Facebook? Or how about your old, old, old Facebook ? Or how about you just go back to MySpace where you get to make all the design decisions, because I remember how good you all were at it. Those MySpace pages were just beautiful. Billions of lines of centred, 18pt, bold Comic Sans in neon yellow on black.

So, you think Facebook have problems ? If Microsoft Office were a country it would be the 3rd largest country in the world, right there after China and India. Close to 500 million users and that’s just the people who’ve bought licenses. It’s also a commercial application which people pay for and on which they depend for their productivity. There are vast communities of “professional” powers users. Hoards of people who use Microsoft Excel for a living. The 5 extra seconds it takes to find one single menu item that’s been moved multiplied by half a billion users means about 100,000 man days of productivity are lost on just the first day a new version of office is released – yeah, yeah, I know. It doesn’t quite work like that, but you get my point.

%d bloggers like this: