SmoothSpan Blog

For Executives, Entrepreneurs, and other Digerati who need to know about SaaS and Web 2.0.

Archive for December, 2007

« Previous Entries

Eventual Consistency Is Not That Scary

Posted by Bob Warfield on December 22, 2007

Amazon’s new SimpleDB offering, like many other post-modern databases such as CouchDB, offers massive scaling potential if users will accept eventual consistency.  It feels like a weighty decision.  Cast in the worst possible light, eventual consistency means the database will sometimes return the wrong answer in the interests of allowing it to keep scaling.  Gasp!  What good is a database that returns the wrong answer?  Why bother? 

Often waiting for the write answer (sorry, that inadvertant slip makes for a good pun so I’ll leave it in place) returns a different kind of wrong answer.  Specifically, it may not return an answer at all.  The system may simply appear to hang. 

How does all this come about?  Largely, it’s a function of how fast changes in the database can be propogated to the point they’re available to everyone reading from the database.  For small numbers of users (i.e. we’re not scaling at all), this is easy.  There is one copy of the data sitting in a table structure, we lock up the readers so they can’t access it whenever we change that data, and everyone always gets the right answer.  Of course, solving simple problems is always easy.  It’s solving the hard problems that lands us the big bucks.  So how do we scale that out?  When we reach a point where we are delivering that information from that one single place as fast as it can be delivered, we have no choice but to make more places to deliver from.  There are many different mechanisms for replicating the data and making it all look like one big happy (but sometimes inconsistent) database, let’s look at them.

Once again, this problem may be simpler when cast in a certain way.  The most common and easiest approach is to keep one single structure as the source of truth for writing, and then replicate out changes to many other databases for reading.  All the common database software supports this.  If your single database could handle 100 users consistently, you can imagine if those 100 users were each another database you were replication to, suddenly you could handle 100 * 100 users, or 10,000 users.  Now we’re scaling.  There are schemes to replicate the replicated and so on and so forth.  Note that in this scenario, all writing must still be done on the one single database.  This is okay, because for many problems, perhaps even the majority, readers far outnumber writers.  In fact, this works so well, that we may not even use databases for the replication.  Instead, we might consider a vast in-memory cache.  Software such as memcached does this for us quite nicely, with another order of magnitude performance boost since reading things in memory is dramatically faster than trying to read from disk.

Okay, that’s pretty cool, but is it consistent?  This will depend on how fast you can replicate the data.  If you can get every database and cache in the system up to date between consecutive read requests, you are sure to be consistent.  In fact, it just has to get done between read requests for any piece of data that changed, which is a much lower bar to hurdle.  If consistency is critical, the system may be designed to inhibit reading until changes have propogated.  It take some very clever algorithms to do this well without throwing a spanner into the works and bringing the system to its knees performance-wise. 

Still, we can get pretty far.  Suppose your database can service 100 users with reads and writes and keep it all consistent with appropriate performance.  Let’s say we replace those 100 users with 100 copies of your database to get up to 10,000 users.  It’s now going to take twice as long.  During the first half, we’re copying changes from the Mother Server to all of the children.  The second half we’re serving the answers to the readers requesting them.  Let’s say we can keep the overall time the same just by halving how many are served.  So the Mother Server talks to 50 children.  Now we can scale to 50 * 50 = 2500 users.  Not nearly as good, but still much better than not scaling at all.  We can go 3 layers deep and have Mother serve 33 children each serve 33 grand children to get to 33 * 33 * 33 = 35,937 users.  Not bad, but Google’s founders can still sleep soundly at night.  The reality is we probably can handle a lot more than 100 on our Mother Server.  Perhaps she’s good for 1000.  Now the 3-layered scheme will get us all the way to 333*333*333 = 36 million.  That starts to wake up the sound sleepers, or perhaps makes them restless.  Yet, that also means we’re using over 100,000 servers too: 1 Mothers talks to 333 children who each have 333 grandchildren.  It’s a pretty wasteful scheme.

Well, let’s bring in Eventual Consistency to reduce the waste.  Assume you are a startup CEO.  You are having a great day, because you are reading the wonderful review of your service in Techcrunch.  It seems like the IPO will be just around the corner after all that gushing does it’s inevitable work and millions suddenly find their way to your site.  Just at the peak of your bliss, the CTO walks in and says she has good news and bad news.  The bad news is the site is crashing and angry emails are pouring in.  The other bad news is that to fix it “right”, so that the data stays consistent, she needs your immediate approval to purchase 999 servers so she can set up a replicated scheme that runs 1 Mother Server (which you already own) and 999 children.  No way, you say.  What’s the good news?  With a sly smile, she tells you that if you’re willing to tolerate a little eventual consistency, your site could get by on a lot fewer servers than 999.

Suppose you are willing to have it take twice as long as normal for data to be up to date.  The readers will read just as fast, it’s just that if they’re reading something that changed, it won’t be correct until the second consecutive read or page refresh.  So, our old model that had the system able to handle 1,000 users, and replicated to 999 servers to handle 1 million users used to have to go to 3 tiers (333 * 333 * 333) to get to the next level at 36 million and still serve everything consistently and just as fast.  If we relax the “just as fast”, we can let our Mother Server handle 2,000 at half the speed to get to 2000 * 1000 = 2 million users on 3 tiers with 2000 servers instead of 100,000 servers to get to 36 million. If we run 4x slower on writes, we can get 4000*1000 = 4 million users with 4000 servers.  Eventually things will bog down and thrash, but you can see how tolerating Eventual Consistency can radically reduce your machine requirements in this simple architecture.  BTW, we all run into Eventual Consistency all the time on the web, whether or not we know it.  I use Google Reader to read blogs and WordPress to write this blog.  Any time a page refresh shows you a different result when you didn’t change anything, you may be looking at Eventual Consistency.  Even if you suspect others changed something, Google Reader still comes along frequently and says an error occured and asks me to refresh.  It’s telling me they relied on Eventual Consistency and I have an inconsistent result.

As I mention, these approaches can still be wasteful of servers because of all the data copies that are flowing around.  This leads us to wonder, “What’s the next alternative?”  Instead of just using servers to copy data to other servers, which is a prime source of the waste, we could try to employ what’s called a sharded or Federated architecture.  In this approach, there is only one copy of each piece of data, but we’re dividing up that data so that each server is only responsible for a small subset of it.  Let’s say we have a database keeping up with our inventory for a big shopping site.  It’s really important to have it be consistent so that when people buy, they know the item was in stock.  Hey, it’s a contrived example and we know we can cheat on it, but go with it.  Let’s further suppose we have 100,000 SKU’s, or different kinds of items in our inventory.  We can divide this across 100 servers by letting each server be responsible for 1,000 items.  Then we write some code that acts as the go-between with the servers.  It simply checks the query to see what you are looking for, and sends your query to the correct sub-server.  Voila, you have a sharded architecture that scales very efficiently.  Our replicated model would blow out 99 copies from the 1 server, and it could be about 50 times faster (or handle 50x the users as I use a gross 1/2 time estimate for replication time) on reads, but it was no faster at all on writes.  That wouldn’t work for our inventory problem because writes are so common during the Christmas shopping season. 

Now what are the pitfalls of sharding.  First, there is some assembly required.  Actually, there is a lot of assembly required.  It’s complicated to build such architectures.  Second, it may be very hard to load balance the shards.  Just dividing up the product inventory across 100 servers is not necessarily helpful.  You would want to use a knowledge of access patterns to divide the products so the load on each server is about the same.  If all the popular products wound up on one server, you’d have a scaling disaster.  These balances can change over time and have to be updated, which brings more complexity.  Some say you never stop fiddling with the tuning of a sharded architecture, but at least we don’t have Eventual Consistency.  Hmmm, or do we?  If you can ever get into a situation where there is more than one copy of the data and the one you are accessing is not up to date, Eventual Consistency could rear up as a design choice made by the DB owners.  In that case, they just give you the wrong answer and move on. 

How can this happen in the sharded world?  It’s all about that load balancing.  Suppose our load balancer needs to move some data to a different shard.  Suppose the startup just bought 10 more servers and wants to create 10 additional shards.  While that data is in motion, there are still users on the site.  What do we tell them?  Sometimes companies can shut down the service to keep everything consistent while changes are made.  Certainly that is  one answer, but it may annoy your users greatly.  Another answer is to tolerate Eventual Consistency while things are in motion with a promise of a return to full consistency when the shards are done rebalancing.  Here is a case where the Eventual Consistency didn’t last all that long, so maybe that’s better than the case where it happens a lot. 

Note that consistency is often in the eye of the beholder.  If we’re talking Internet users, ask yourself how much harm there would be if a page refresh delivered a different result.  In may applications, the user may even expect or welcome a different result.  An email program that suddenly shows mail after a refresh is not at all unexpected.  That the user didn’t know the mail was already on the server at the time of the first refresh doesn’t really hurt them.  There are cases where absolute consistency is very important.  Go back to the sharded database example.  It is normal to expect every single product in the inventory to have a unique id that lets us find that part.  Those ids have to be unique and consistent across all of the shards.  It is crucially important that any id changes are up to date before anything else is done or the system can get really corrupted.  So, we may create a mechanism to generate consistent ids across shards.  This adds still more architectural complexity.

There are nightmare scenarios where it becomes impossible to shard efficiently.  I will over simplify to make it easy and not necessarily correct, but I hope you will get the idea.  Suppose you’re dealing with operations that affect many different objects.  The objects are divided into shards naturally when examined individually, but the operations between the objects span many shards.  Perhaps the relationships between shards are incompatible to the extent that there is no way to shard them across machines such that every single operation doesn’t hit many shards instead of a single shard.  Hitting many shards will invalidate the sharding approach.  In times like this, we will again be tempted to opt for Eventual Consistency.  We’ll get to hitting all the shards in our sweet time, and any accesses before that update is finished will just live with inconsistent results.  Such scenarios can arise where there is no obvious good sharding algorithm, or where the relationships between the objects (perhaps its some sort of real time collaborative application where people are bouncing around touching objects unpredictably) are changing much too quickly to rebalance the shards.  One really common case of an operation hitting many shards is queries.  You can’t anticipate all queries such that any of them can be processed within a single shard unless you sharply limit the expressiveness of the query tools and languages.

I hope you come away from this discussion with some new insights:

–  Inconsistency derives from having multiple copies of the data that are not all in sync.

–  We need multiple copies to scale.  This is easiest for reads.  Scaling writes is much harder.

–  We can keep copies consistent at the expense of slowing everything down to wait for consistency.  The savings in relaxing this can be quite large.

–  We can somewhat balance that expense with increasingly complex architecture.  Sharding is more efficient than replication, but gets very complex and can still break down, for example. 

–  It’s still cheaper to allow for Eventual Consistency, and in many applications, the user experience is just as good.

Big web sites realized all this long ago.  That’s why sites like Amazon have systems like SimpleDB and Dynamo that are built from the ground up with Eventual Consistency in mind.  You need to look very carefully at your application to know what’s good or bad, and also understand what the performance envelope is for the Eventual Consistency.  Here are some thoughts from the blogosphere:

Dare Obasanjo

The documentation for the PutAttributes method has the following note

Because Amazon SimpleDB makes multiple copies of your data and uses an eventual consistency update model, an immediate GetAttributes or Query request (read) immediately after a DeleteAttributes or PutAttributes request (write) might not return the updated data.

This may or may not be a problem depending on your application. It may be OK for a del.icio.us style application if it took a few minutes before your tag updates were applied to a bookmark but the same can’t be said for an application like Twitter. What would be useful for developers would be if Amazon gave some more information around the delayed propagation such as average latency during peak and off-peak hours.

Here I think Dare’s example of Twitter suffering from Eventual Consistency is interesting.  In Twitter, we follow mico-blog postings.  What would be the impact of Eventual Consistency?  Of course it depends on the exact nature of the consistency, but lets look at our replicated reader approach.  Recall that in the Eventual Consistency version, we simply tolerate that we allow reads to come in so fast that some of the replicated read servers are not up to date.  However, they are up to date with respect to a certain point in time, just not necessarily the present.  In other words, I could read at 10:00 am and get results on one server that are up to date through 10:00 am and on another results only up to date through 9:59 am.  For Twitter, depending on which server my session is connected to, my feeds may update a little behind the times.  Is that the end of the world?  For Twitter users, if they are engaged in a real time conversation, it means the person with the delayed feed may write something that looks out of sequence to the person with the up to date feed whenever the two are in a back and forth chat.  OTOH, if Twitter degraded to that mode rather than taking longer and longer to accept input or do updates, wouldn’t that be better? 

Erik Onnen

Onnen wrote a post called “Socializing Eventual Consistency” that has two important points.  First, many developers are not used to talking about Eventual Consistency.  The knee jerk reaction is that it’s bad, not the right thing, or an unnecessary compromise for anyone but a huge player like Amazon.  It’s almost like a macho thing.  Onnen lacked the right examples and vocabulary to engage his peers when it was time to decide about it.  Hopefully all the chatter about Amazon’s SimpleDB and other massively scalable sites will get more familiarity flowing around these concepts.  I hope this article also makes it easier.

His other point is that when push comes to shove, most business users will prefer availability over consistency.  I think that is a key point.  It’s also a big takeaway from the next blog:

Werner Vogels

Amazon’s CTO posted to try to make Eventual Consistency and it’s trade offs more clear for all.  He lays a lot of good theoretical groundwork that boils down to explaining that there are tradeoffs and you can’t have it all.  This is similar to the message I’ve tried to portray above.  Eventually, you have to keep multiple copies of the data to scale.  Once that happens, it becomes harder and harder to maintain consistency and still scale.  Vogels provides a full taxonomy of concepts (i.e. Monotonic Write Consistency et al) with which to think about all this and evaluate the trade offs.  He also does a good job pointing out how often even conventional RDMS’s wind up dealing with inconsistency.  Some of the best (and least obvious to many) examples include the idea that your mechanism for backups is often not fully consistent.  The right answer for many systems is to require that writes always work, but that reads are only eventually consistent.

Conclusion

I’ve covered a lot of consistency related tradeoffs involved in database systems for large web architectures.  Rest assured, that unless you are pretty unsuccessful, you will have to deal with this stuff.  Get ahead of the curve and understand for your application what the consistency requirements will be.  Do not start out being unnecessarily consistent.  That’s a premature optimization that can bite you in many ways.  Relaxing consistency as much as possible while still delivering a good user experience can lead to radically better scaling as well as making your life simpler.  Eventual Consistency is nothing to be afraid of.  Rather, it’s a key concept and tactic to be aware of.

Personally, I would seriously look into solutions like Amazon’s Simple DB while I was at it. 

Posted in amazon, data center, enterprise software, grid, platforms, soa, software development | 6 Comments »

March of the Sugar Plum Commoditizers: Maybe It’s A Sign There’s Less Bubble Than Thought?

Posted by Bob Warfield on December 20, 2007

No, I’m not calling commoditizers fairies are even trying to be derogatory, it was just a title that seemed to work at this time of the year.  What it’s about is that we’re seeing some fruits of the Web 2.0 tree ripening enough that the commoditizers see an opportunity.  What does a commoditizer do?  They follow the innovators in the food chain and ensure broader availability of the innovation.  That’s the commoditizer elevator pitch.  Typically they do this in at least four ways:

– They offer alternatives to the Innovator’s product, thus ensuring competition.

– They undercut the Innovator’s pricing, which leads to broader availability for all.

– They enable white label versions of the innovation, which essentially lets many others use the innovation as their own platform.

– They broaden the category to niches that may have some special requirements.

This is good stuff in a market.  It reflects that enough value has been recognized around an innovation that it’s time to take it to the next stage.  The Innovators will frequently view the Commoditizers as late-to-the-party freeloaders who just want to cash in on someone else’s idea.  In part they are correct, yet the broader markets and user bases benefit greatly by not having an idea totally locked up in the Innovator’s hands.

So who are the current crop of commoditizers commoditizing?

– Ning hit the Facebook/MySpace world.  They’re in my third and fourth categories as white labelers who are also catering to niches.

– Fraxi is trying to commoditize the world of Digg, just at the time when Digg seems to have stopped growing so much and is evidently trying to sell itself.  Fraxi isn’t out yet, but looks like a white label play.

– Video seems to be going crazy with commoditization.  Everywhere you look someone is announcing a new video service or adding video to their existing service.  This may presage a broader commoditization of media on the web.  Consider the willingness of the TV industry to experiment with it after watching the fate of their record company brethren.  RadioHead made news and shows the future here by offering their latest album and letting the buyers decide the price.  According to the band they made more money out of that release than all of their prior releases combined.  Companies like Scoble favorite Kyte (just announced a $15M Series B) are raking in the VC.  We’ve heard all about Seesmic and now there’s a company called Magnify that may be trying to steal some of their thunder.  Yep, video is definitely in the commoditization phase.

– Even in cloud platform land, a relatively new area, commoditizers have shown up.  Intel is staking Nirvanix, which offers an alternative to the Amazon S3 storage service.  I might add that the whole Amazon approach is more or less aimed at commoditizing data centers and hosting.

There are lots of other good examples kicking around out there in your RSS feeds. 

What does it all mean?

First, it’s clearly a good thing in the short run.  It means more choices and lower prices for customers.  That’s always a good thing to inject into a market.  Second, and this latter is much more speculative, it is a sign we might be involved in a little less bubble than may have been thought.  How can I possibly say that in the wake of $15B Facebook valuations and all the rest?  The short answer is that I can’t with any certainty, but we’ve entered the litmus test phase for these areas.  The initial successes are there.  Facebook, MySpace, YouTube, and even Digg all count.  We’re now in the phase of seeing if the commoditizers can make a go of it.  If some of them start to be successes in the same way the Innovators have, that’s how we’ll know we’ve got real markets and not just a bubble.  The reason is that the only way commoditizers survive is that they open things up to moving the market across the chasm.  Yes folks, you can’t support a ton of commoditizers with early adopters.  Commoditization requires real demand.  Real success for that category means we’re past the early adopter stage and a much more difficult and much larger market has been engaged.

We could still falter, especially with the economy the way it is, but I see some world changers out there, and I’m rooting for the commoditizers to take hold.  The other benefit of all this is that eventually it will signal that it’s time for another punctuated equilibrium and a new set of innovators will be empowered to bring us the next wave.  It’s hard for that process to get underway until the world has digested the current innovation crop and has room to look at something genuinely new.

Posted in saas | Leave a Comment »

Just OpenID’d My Way to Blogger

Posted by Bob Warfield on December 20, 2007

Hey cool, I just had my first encounter with OpenID.  I was writing a comment over on Aloof Architecture and it offered me a chance to sign in with my WordPress account.  Worked like a charm and it felt great!

Kind of like being shown into a restaurant ahead of others because the Maitre ‘D knows you.

Let’s see more of this, it’s the way the web ought to work.

For the record, it asked for my WordPress blog name.  It then kicked me over to WordPress (which I was already logged into) so I could confirm it was okay to login to Blogger, and then I was done.  I had to option to approve for just the one login or permanently.

Just one more thing made simpler in life.

Posted in Web 2.0 | Leave a Comment »

Green DC for Datacenters, Why Not Homes Too?

Posted by Bob Warfield on December 19, 2007

There are some posts up about how much greener DC power is for datacenters.  Phil Windley says a single server may have a larger carbon footprint than a 15 mpg SUV.  That was an amazing statistic for me to hear.  CNet reports there is a startup called Validus that is working on the idea of supplying DC within datacenters instead of AC to help combat the waste.

The idea is that converting AC to DC is wasteful of energy, particularly when it is done over and over again inside every digital box.  Whether we’re talking wall worts for small boxes or full-on PC power supplies (or worse) for bigger boxes, it is wasteful.  I don’t find that hard to believe.  All these power supplies get warm to the touch.  Heck, my PC supply has a big fan built right into it.  This leads me to ask:  If this is a good idea for datacenters, why isn’t it a good idea for homes too?

I wonder what percentage of the power we use in our homes would be better off DC rather than AC?  Most electronics immediately want to convert to DC.  All of our computers and home entertainment electronics are in that category.  Many things with motors are not currently in that camp, but they could be with DC motors.  I’m speaking of refrigerators, for example.  It’s not clear to me that something like an electric range or water heater really cares one way or the other.  Certainly anyone that has built a PC can see that it would be easy enough to give up the power supply.  The principle issue is simply providing cabling with the right voltages.  It’s the same cabling that comes out of your power supply today.  One issue would involve those voltages.  It’s very easy to change voltages in the AC regime with a simple transformer.  DC is more problematic.  Unfortunately, even PC’s like several different voltages to be available.  A typical PC power supply offers -12V, -5V, 0V (ground), 3.3V, 5V, and 12V.  Phew, that would be a lot of outlets on the wall!  I wonder what the chances are we could get by on fewer voltage options.

The other interesting thing is that DC is ideally suited for batteries and solar power.  As it stands, those who wish to employ solar power have to run the DC output from their solar panels into an inverter to convert it to AC.  Of course this costs energy through losses to make the transformation.  If we had more of a DC-based economy, life would be better.

So why doesn’t the power company just provide DC instead of AC?  There was an epic battle fought between Thomas Edison, a DC proponent, and Nikola Tesla, the father of many early AC inventions, that Edison ultimately lost.  It turns out to be cheaper to generate AC with rotating machinery, and cheaper to transmit it over long distances.  If we’re going to go DC, we need to do so at the “last mile”.  It remains an intriguing idea.  I’ll be on the lookout for an “appliance” power supply that will run multiple computers and save power.

Posted in data center | 5 Comments »

Making Enterprise Software Sexier: Repeatable Process Without Endless Boredom

Posted by Bob Warfield on December 19, 2007

I couldn’t quite get past thinking about the issue of “sexy Enterprise Software” that Scoble started us talking about a few days ago.  The topic was such a Rorschach Test for what people think of when they think Enterprise Software.  That in and of itself was quite interesting.  Seeing lots of new perspectives on a thing I thought was familiar is always food for thought.  After a few days of almost unconscious cogititation, an interesting new perspective or two often starts to take root.  My initial response to it all was to suggest that what was interesting about the software was perhaps not so much the immediate UI as the potential for business transformation that Enterprise Software offers, but there were wildly differing perspectives.  Michael Krigsman and Nick Carr got into a real snit over the whole thing.  Nick Carr, who often writes that IT is becoming irrelevant, was strangely passionate about the idea that there is nothing precluding business software from being engaging, while Krigsman was determined that it’s a complete non sequitor to even ponder the sexiness of Enterprise Software because it’s all about repeating a crucial business process:

I demand the system work every time without fail.

While I disagree with much of what Krigsman had to say (including his parting shot at Carr which asks where’s the revenue: compare Google and Oracle and you’ll see the revenue is very much there), the idea that it was all about perfecting process got me thinking. 

Stowe Boyd tickled my thinking further, when he says that Enterprise Software is about Groups not Individuals:

Enterprise software starts with the premise that the user is an employee, or member of the marketing department, or a minion in the IT department. The users rights and capabilities are tied to membership, not to individual identity.

Stowe gets a lot closer to the way I am thinking about making Enterprise software sexier, but not all the way.

My new thinking is focused around the idea that a lot of Enterprise Software is about implementing Repeatable Process.  Perhaps the ultimate incarnation and priesthood for Repeatable Process is Six Sigma.  Practitioners will point out that there is a lot more to it than repeatability, in the sense that one is supposed to continually improve the process.  Any defect that is identified results in a process change that eliminates the root cause of the defect so it can’t happen again.  It can be a powerful methodology: much of the success for companies like General Electric is laid at the alter of Six Sigmas. 

This is all fine and well, but I think it can be taken too far and applied in a manner such that the endlessly repeated process results in endless boredom.  That in and of itself can be a problem, because boredom leads to mistakes and lack of attention.  Boredom costs productivity, which is not what we’re about with Enterprise Software.  There is a further danger to Six Sigmas that has started to be heard more recently.  According to Wikipedia:

A Fortune article stated that “of 58 large companies that have announced Six Sigma programs, 91 percent have trailed the S&P 500 since.” The statement is attributed to “an analysis by Charles Holland of consulting firm Qualpro (which espouses a competing quality-improvement process).”[15] The gist of the article is that Six Sigma is effective at what it is intended to do, but that it is “narrowly designed to fix an existing process” and does not help in “coming up with new products or disruptive technologies.” Many of these claims have been argued as being in error or ill-informed.[16][17]

When Six Sigma is used as a cost cutting program, it has been shown to stifle new product innovation.[18]

Of course this gets the High Priesthood into a fine frothy fit, but it sure seems to fit with everything I know and have read about innovation.  Even Deming advocates something shockingly similar to the choices I’m suggesting for our Enterprise Users.  According to Wikipedia, one of his principles is Semi-Automated, not Fully Automated:

Dr. Deming lamented the problem of automation gone awry (“robots painting robots”): instead, he advocated human-assisted semi-automation, which allows people to change the semi-automated or computer-assisted processes, based on new knowledge. Compare to Japanese term ‘jidoka’ (which can be loosely translated as “automation with a human touch”).

I like this concept of semi-automation, where everyone understands the ultimate goals and destination, but there is personal freedom about how best to get there.  If Deming, the Grand Pajandrum of consistent process execution can advocate choice, why doesn’t it make sense here?

What then to do about it?  In part, I think there is a problem in that we continue to carry over a lot of the old 3270 Green Screen approach to Enterprise Software.  We view that the job of the computer is to rigidly control the people using it and that this is the only way to ensure our perfectly concieved process will be accurately carried out.  Much of it has extremely rigid workflow.  Such user interfaces are extremely modal, and frankly, they feel awful.  They force their users to act in a pre-defined play like marionettes.  There are endless “screens” that have to be dealt with, and sometimes a key piece of information is on a screen that we can’t quite remember how to get to.  The track is extremely rigid, and usually make no sense at all to a newcomer.  It might also be less than ideal for achieving the best possible results.  After all, people are not marionettes.

3270 Green Screen

Jason Kolb says this about Krigsman’s posts:

…his point is that enterprise software is to be stable, above all else, sacrificing the user if need be. 

Jason is not advocating this, in fact quite the opposite, but it is another way of saying what Enterprise Software is in many cases.  I take exception to the idea that we may need to sacrifice the user in favor of the process.  People are not at their best and most productive when treated this way.  There is a tremendous amount of literature about the value of offering people choices as they go about their tasks, but the rigid process brand of enterprise software doesn’t hold with that.

An awful lot of Enterprise Software is nothing more than moving database records and fields from one form to the next while some poor user is expected to do some menial task to the form that the computer can’t quite manage for itself.  Shades of Amazon’s Mechanical Turk.

What if there is a better way?  What might it be?

There are tantalizing glimpses out there.  One of the best comes from James Governor, who made a long post about some SAP demos he has seen.  It’s worth a read.  The essence of it is enabling the ad hoc to become a part of the process.  I think of it as self-modifying process.  It is modified if and when the users of the process need to change it to do better.  Such an idea places individuals back into control, at least a little bit.  After all, part of Six Sigma involves continuously improving the process to make it better.   When was the last time your Enterprise Software was malleable enough to improve on the fly?  I thought so.  Yet here is an example that shows how to give at least some of the users choices rather than locking them into an immutable process.

Perhaps it is simply a matter of providing choice that matters to the users about what to do next.  Choice that goes beyond the green screen menu.  It blinks at you and says smugly, “Figure out how to fit what you want into my world or you’ll never get there.”  No, we want software that fades into the background.  It has choices that are natural, make more sense to people than machines, and that any reasonable human being can fit together in the way that people are great at and computers are lousy at to make anything they need to have happen get done.  One of the maddening things about “workflow” and “business process” is that it’s so synchronous.  You have to follow a task to its completion or lose everything.  It’s like those annoying web storefronts where if you make one tiny little mistake on a form late in the process, it kicks you back and makes you reenter all of the information. 

Such choices might be more like Google’s system of project management.  They’re a very textual company, so they use a system based on email.  Folks receive emails asking them what they expect to accomplish next.  They respond via email.  The system then comes back at a later date and asks whether they got the tasks done.  Reporting based on these results is available to managers.  Wouldn’t such a system be tremendously more friendly than trying to fight through Microsoft Project or some other highly stylized project management system?  Given all the complaints about Sales Force Automation (salespeople often hate filling in the data), it made me wonder why you couldn’t create a CRM system around the same principles.  Maybe that would be the ultimate killer CRM that salespeople would finally accept as being worthwhile.

I wonder when the last time a workflow designer (Business Process Architect or whatever high-faluting title they use) actually thought about giving users more choices was?  Hmmm, probably depressing to consider.

Maybe users would be happier if the tasks were much more granular and the users could bounce around as they saw fit.  How do you maintain the integrity of a process in a world like that?  Well, it’s an asynchronous world, so you have to keep track of what’s left to do and let the users decide what to do next.  The email inbox. RSS feed, and Social Network newsfeed are such paradigms.  They’re worlds where users can open as many cans of worms as they like and work on them in any order.  It recognizes that people are people.  They get interrupted.  They change their minds.  They may forget or need to go back to something.  They get tired of doing one thing and need to do another for a while as a respite.  But despite all of that, they don’t want to be led around by the nose.  They want choice.

With choice comes responsibility.  Folks are accountable for their choices.  That’s okay.  Responsibility is a good thing.  People rise to the challenge more often than not and everyone is better for it.  But we need to help people make the best choices.  Give them feedback.  Give them a way to keep score on their choices.  Some form of analytics.  To choice and analytics, let’s add another valuable spice: collaboration. Consider a case that seems completely unrelated:  in Agile programming, a lot of productivity is gained by having programmers work in pairs instead of as individuals.  Collaboration is not only fostered, it is forced.  And it turns out to be a lot of fun once you get used to it.  Does anyone do something like that with Enterprise Software?  Is there no way to pair up users and benefit?  I’ll bet there must be.

Here is another crazy thought: what if we need to turn Enterprise Software inside-out to make it sexy?  Old school Enterprise has various user categories that are largely clerks locked into precise workflows based on how a particular company likes to operate.   HR and Finance are the categories where most of this stuff lives.  Enterprise Software has succeeded in making the clerks wildly more efficient, so we need fewer of them, but there are still many many clerks out there.  How can we turn Enterprise Software inside-out and eliminate the need for clerks?  Remember (or perhaps you don’t, there are fewer and fewer who do) how it used to be before the PC was common and there was such a thing as a Knowledge Worker?  Managers all had assistants who did things like taking memos and writing letters.  Go further back and we learn that CC meant “Carbon Copy”, which was literally the process used to send a memo to more than one person.  Word Processors on PC’s turned that task inside out in a relatively short period of time and there was suddenly much less need for everyone to have an assistant.  Now an assistant could take care of a relatively large department, and he could do so with day-to-day tasks that were at a much more interesting level. 

And then I had the craziest idea yet.  What if the software was designed in a way that its users actually wanted to follow the best process?  What if they actually enjoyed it, or perhaps at the least overlooked that it was repetitive and boring for some reason?  Think about it for a minute.  Isn’t most of what goes on with Social Networks repetitive and boring if you strip it down and just look at what you can do?  All that newsfeed reading and poking and messaging, it isn’t as if this is the world’s greatest game.  Yet people seem to stay engaged.  Some would say addicted even.  What’s needed here?  An incentive would be my first thought.  A reason to move forward and do the next thing well.  A reason that is more compelling, more immediate, and more under the user’s control than just the promise of their next paycheck.  Incentives are a powerful force, and often, they need not even be monetary to be effective.  I mentioned the idea of keeping score above.  I’ve used score keeping many times to get competitive people working above and beyond the call of duty.  It’s a way of life for salespeople, and they love it.  It’s tantamount to turning work into a game, albeit with more serious consequences.

Let me give one last example that ties all of this together.  There is more choice, there is scorekeeping, there is self-service, and at the end of the day everyone benefits.  A friend recently suggested a great application for businesses that have lots of part-time help.  My brother manages a retail store, and his situation is not unique.  He has available 90 part-time hours for the store, and 5 or 6 employees to spread those hours around to.  Nobody is ever happy about it.  They all fight constantly for more hours, and this places my brother as manager into the role of the “heavy” who has to say “no” more often than “yes.”  You can just imagine some heavy-handed Enterprise Software arbitrating this fight with an obscure HR-derived algorithm that is deemed fair to all but annoys everyone and is almost opaque to understanding.

Enter this other friend’s idea.  He wants to turn the whole thing into a Dutch Auction.  People would hop onto a web application and bid what hourly rate they’re willing to take for a certain number of hours.  Lower rates get more hours.  He sees this as a win-win.  Management is no longer cast in the role as heavy.  If you an employee really wants a lot of hours, they can bid to get them.  Chances are that a slightly lower hourly rate is more than offset in gross dollars by gaining more hours.  The company was going to pay for 90 hours (in my brother’s case) anyway, so it wins by getting a lower hourly rate over those hours.  Management is happy because they’re no longer in the middle of a no-win dispute.

It’s a case of self-service, offerings choices, and loosening up a process until people can participate and imporve the situation for everyone.  By completely rethinking the problem instead of paving the cowpaths (i.e. automating existing manual processes), it’s possible to provide radically better solutions.  Enterprise Software can be sexy, but only if you think of the people more than the process.

Posted in business, enterprise software, saas, strategy | 6 Comments »

What if Twitter Was Built on Amazon’s Cloud?

Posted by Bob Warfield on December 18, 2007

There was recent bellyaching in the blogosphere again about Twitter being down.  Dave Winer grumbles, “What other basic form of communication goes down for 12 hours at a time?”  There are various comments, and in the end, apparently it was about their moving ISP’s.  Twitter themselves had this to say:

Twitter is humming along now after a late night. Our team worked earnestly into the night and morning on our largest and most complex maintenance project ever. Everything went pretty much according to plan except for one thing: an incorrect switch.

The switch in question caps traffic an unacceptable level. In order to correct this, we’ll need to get some hardware installed. Unfortunately, that means we’re not done with our datacenter move just yet. This type of work can be frustrating but it’s all towards Twitter’s highest goal: reliability.

Such moves are never easy, they always include a hitch of some kind, and the Twitter customer base is hopelessly addicted to the medium so Twitter hears about it whenever the turn the thing off for any period of time.  I look at this and for me it’s just one more reason I wouldn’t want to own a datacenter.

Suppose your service, or maybe even Twitter, was built on Amazon’s Cloud or some other Utility Computing solution.  You don’t own the servers, you are renting them.  If loads go up, you can simply rent more in direct proportion to the loads and on 10 minutes notice.  A recent High Scalability article on scaling Twitter shows they don’t really have all that many servers:

  • 1 MySQL Server (one big 8 core box) and 1 slave. Slave is read only for statistics and reporting.
  • 8 Sun X4100s.

    • 10 boxes, in other words.  Now it comes time to upgrade.  Much pain and frustration.  To do it well, and without interruption, they really need 2 complete copies of their infrastructure.  This way, they can prepare the new version and start cutting users over to it while leaving the old one running.  When everyone is over, the old system can be decommissioned.  For many startups, owning twice as much hardware as they use is just out of the question.  The more successful they become, the more expensive it becomes to entertain such a luxury.  Not so on a utility computing service like Amazon’s.  Purchase the use of twice as many servers for just how long it takes for a successful upgrade and then cut them loose afterward.

    There are detractors to the Amazon approach out there, but do we really think it would make Twitter much less reliable?  What if it made it much more reliable?

    Here’s another thought that runs rampant:  how well would Amazon’s new SimpleDB work for a service like Twitter?  It seems tailormade.  Certainly the notion of a “texty” database with up to 1024 characters per field seems like a fit.  It would be fascinating to see some of the Twitterati put up a Twitter clone on Amazon’s Web Services using SimpleDB just to see how well it works and how quickly it could be put together.  Given the platform and the requirements of the application, it seems like it would not be that hard to do the experiment.  It would certainly make for an interesting test of how well Amazon’s infrastructure really works.

    Posted in data center, ec2, grid, platforms, Web 2.0 | 2 Comments »

    Intelligent Agents? Show Me the Demo!

    Posted by Bob Warfield on December 18, 2007

    I’ve been hearing about Intelligent Agents for years.  They’re supposed to be capable of revolutionizing our PC experience by automating away all sorts of drudgery.  Exactly what an Intelligent Agent might be is never completely clear.  We’re given a few simple examples and a lot of mumbling about how great it’s going to be when computers are powerful enough to run them in 5 or 10 years.  Microsoft Chief Research Officer Craig Mundie is just one of the latest to trot out this old chestnut at their shareholder meeting.  I got it courtesy of Greg Linden, and one of Mundie’s remarks sparked this post:

    We’ve done this at the level of speculative execution … but only for the purposes of trying to make the machine go faster … Can … software that is wildly more complex and sophisticated … but well suited to this class of machine that will emerge in the next 5 to 10 years … make [machines] qualitatively different and better … [and] make the machine something that really borders on being your [assistant]?

    To this 5 to 10 year business I say, show me the demo!  Think about it.  If 5 to 10 years of further PC performance development is what’s needed, why can’t we see the demo today?  Run a cluster of PC’s to deliver the demo.  Run them on Amazon’s Web Services.  What are we talking about here?  Let’s keep to the (sometimes) optimistic Moore’s Law and assume we’ll have twice as many cores every 18 months.  Today we get 4 cores.  Here is our timetable:

    – 1.5 years from now:  8 cores = 2 PC’s of today

    – 3 years from now 16 cores = 4 PC’s of today

    – 4.5 years from now 32 cores = 8 PC’s of today

    – 6 years from now 64 cores = 16 PC’s of today

    – 7.5 years from now 128 cores = 32 PC’s of today

    – 9 years from now 256 cores = 64 PC’s of today

    – 10.5 years from now 512 cores = 128 PC’s of today

    So, a cluster of 128 PC’s allows us to simulate the power of a PC 10 years hence.  This is certainly well within the capacity of the Microsofts and Googles to run such simulations.  Heck, using something like Amazon EC2 its within the power of a startup or university to play with this kind of thing.  So let’s quit talking about how good it’s going to be and SHOW ME THE DEMO!

    Greg is not quite so starry eyed as to chalk this up as merely a matter of waiting for enough PC power.  He points out that knowing what software to write is essential too.  Certainly some of the attempts at intelligent agents so far have been at best laughable and at worst extremely annoying (Die, Clippy, die!).  Wouldn’t it be exciting to see an actual demo where the computer was doing something scary useful along the lines of intelligent agents?  Wouldn’t it be great to be the company pioneering all that stuff and no doubt patenting every detail along the way?

    Part of the challenge is in knowing what the software should do.  Part of it is in amassing enough data to do it.  There was recently a fascinating blog post in the Google Operating System Blog that talks about the value of large amounts of data.  Apparently, Google is working on improving search on videos by enabling speech recognition of the video sound track.  That’s a cool idea, I have to admit.  To get there, they needed a lot of data on how people speak various phonemes.  So, they created the Google 411 service to collect that data.  This is a very clever move that is important when dealing with this kind of problem.  They chose to solve a very narrow problem so that they could learn from the problem.  Think about what a 411 service must do.  It’s a pretty limited problem and they understand both sides of the conversation when its working.  Understanding both sides is the key to building a bridge between them.

    It will be much the same with intelligent assistants.  The first problem is simply to understand what this assistant will do.  Nobody seems to have a unified theory of it: we get dribs and drabs of examples that often seem unrelated.  Many of these examples are going to be very squishy.  There is no quick and clean algorithm for what must be done.  Lots of heuristics will be called into play.  This is not unlike speech recognition, which started out recognizing phonemes and when that still didn’t work, they added statistical likelihood.  What word is most likely to follow the prior word?  What happens if we stack up those probabilities to augment words we clearly recognize versus words we’re having a hard time with?  Now let’s feed in grammar and parts of speech.  Each addition moves us a little further, but there is no simple solution.  And data will be hugely valuable, even essential, to gaining that understanding.

    These two insights, the importance of data and the do-ability of assembling a powerful enough demo platform, make me wonder about whether Intelligent Agents won’t turn out to be more of a cloud computing thing than a desktop thing.  Consider that whatever these agents are going to do, if it can be pre-processed to a large degree, then they only need to keep up with data as it changes.  Across the entire web, a huge amount of data is changing.  On a single PC, not so much data.  My household has 5 computers that could be potential consumers of the Intelligent Agent service.  We backup all of these machines to a NAS drive on the LAN, yet, the amoung of change as evidenced by backup logs is pretty slight.  We could surely share that 128 PC cluster with a number of other households before it got too slow.  After all, the cluster could never have spent all its time intelligent agenting even in Mundie’s scenario lest the underlying computer be completely unusuable.  If we are using a separate cluster, it’s okay to completely monopolize it.

    Isn’t the Cloud ideal for this kind of thing?  The purveyors that live within the cloud have the data and the huge numbers of machines to deliver such a thing.  Being able to receive such benefits might be one more reason to move your data into the cloud.  I won’t hold my breath.  I’ve heard about Intelligent Assistants too many times, but I do think it’s time to demand the demo rather than sit back and listen to another blue sky prediction.

    Posted in Web 2.0 | Leave a Comment »

    Microsoft Has Played the Role of Disruptor, and Disrupted…

    Posted by Bob Warfield on December 16, 2007

    The NYT has a nice piece called “Google gets ready to rumble with Microsoft.”  The fighting words in question are Eric Schmidt’s prognostication that if customers could wave a magic wand, they’d be better off with 90% of their computing in the cloud and 10% on their desktop PC’s.  Obviously Microsoft has to take exception to this, since that would eliminate a huge amount of their business.  One the chief desktop proponents at Microsoft would be Jeff Raikes, whose response indicates he feels Google’s remarks are pretty self-serving:

    “It’s, of course, totally inaccurate compared with where the market is today and where the market is headed,” says Jeff Raikes, president of Microsoft’s business division, which includes the Office products.

    Yeah, whatever.  The NYT wound you guys up and set you at each other’s throats like a couple of pitt bulls.  It’s a time honored tradition among reports, and besides, they spelled everyone’s name right, so it’s good PR in the end.  Yet there are some nagging doubts out there.  What if the Google guys are right? 

    Ask yourself this question:  Is there any new desktop software being created today, or is all the innovation happening on the web?  At best I think we’re seeing the new breed of web apps that live well on the desktop courtesy of Google Gears and Adobe AIR.  This puts a shelf life on the dreams of the desktop empire builders.  They have a period of time in which to get their act together and learn to be the top dogs in the clouds.  Don Dodge’s message is not to count Microsoft out because:

    It is important to remember that Microsoft has played the role of “Disruptor” in the past, and learned a lot along the way.  

    Color me skeptical.  Microsoft has been at the Internet game for a long time now, since the days of fighting it out with Netscape.  The outcome of that battle was that Microsoft was the disruptee, not the disruptor.  They thought the Internet was all about the app, so they have huge browser share that doesn’t really matter strategically.  Meanwhile the web went along on its own way and Microsoft’s participation in building out the big web companies has been sharply limited as I’ve written before.  I don’t see the web so far as a proud chapter of successful Microsoft plays so much as a time when they’ve struggle breathlessly in race after race only to learn the finish line moved on before they could get there.  Has Microsoft learned a lot?  Perhaps.  Maybe Ray Ozzie brings that new perspective to their camp. 

    OTOH, this may turn out to be disruption that is similar to SaaS.  It’s extremely hard for a company used to selling perpetual licenses to convert over to SaaS.  The business model is corrosive to their financial results for one thing.  Another issue is that it changes almost every aspect of how they think about doing business.  Is it so far fetched to think that a totally advertising oriented model like Google’s could be very corrosive to Microsoft’s history of charging for software license and historically Kingly margins? 

    Dan Farber points to an interesting insight, which is that Google’s massively dominant search position has an impact on the other areas.  What if you could easily access search to manage all of your information?  Microsoft had been promising a better answer than file folders for a long time.  It was supposed to be part of Vista,  but we are still waiting.  This is definitely a point of vulnerability.  When does powerful search, collaboration, freedom from manually shifting data between machines, freedom from installing or upgrading, and all the rest finally add up to enough that people move in droves to the new model?

    This is one of those things where you’ll wake up and the revolution will be well underway.  Suddenly you’ll notice a lot of friends have already crossed the bridge and are waving back at you to join them.  Think of the MP3 revolution.  The awesome power of free struck the record companies with a vengeance.  Even though the sound quality was very poor compared to good CD playback, people started switching in droves at some point and never looked back.  Consider the Macintosh.  For years it was written off as dead.  Now if you don’t have fruit on your machine you’re just not hip.

    Microsoft is one of the most (if not THE most) hypercompetitive companies on the planet.  I don’t think Google has any idea what they’re dealing with there.  Google is not hypercompetitive.  They’re great technologists who’ve been in the right places at the best possible times and they occupy the strategic high ground in a way that makes them the Microsoft in this new battle.  It will be interesting to watch the contest unfold.  Microsoft had just better not wait too long to figure out the winning plays.

    Posted in strategy, Web 2.0 | 10 Comments »

    To Rule the Clouds Takes Software: Why Amazon SimpleDB is a Huge Next Step

    Posted by Bob Warfield on December 15, 2007

    One Ring to rule them all, One Ring to find them,
    One Ring to bring them all and in the darkness bind them…

    J. R. R. Tolkien

    There is much interesting cloud-related news in the blogosphere.  Various pundits are sharing a back and forth on the potential for cloud centralization to result in just a very few datacenters and what that might mean.  The really big news is Amazon’s fascinating new addition to their cloud platform of SimpleDB.  Let’s talk about what it all means.

    Sun’s CTO, Greg Papadopoulos, has been predicting that the earth’s compute resources will resolve into about “five hyperscale, pan-global broadband computing services giants” — with Sun, in its version of this future scenario, the primary supplier of hardware and operations software to those giants. The last was channeled via Phil Wainewright, who goes on to ask, “What is it about a computing grid that’s inherently “more centralized” in nature?”  He feels that Nick Carr has missed the mark and swallowed Sun’s line hook, line, and sinker.  For his part, Carr’s only crime was to seize on a good story, because at the same dinner, another Sun executive, Subodh Bapat, was telling Carr that sometime soon a major datacenter failure would have “major national effects.”  The irony is positively juicy with Sun talking out both sides of their proverbial mouths.

    The tradeoff that Carr and Wainewright are worried about is one of economies of scale that favor centralization versus flexibility and resiliency that favors decentralization.  Where they differ is that Carr sees economies of scale winning in a world where IT matters less and less and Wainewright favors the superior architectural possibilities of decentralization.  Is datacenter centralization inexorable?  In a word, yes, but it may not boil down to just 5 data center owners, and it may take quite a while for the forces at work to finish this evolution.  The factors that determine who the eventual winners will be are also quite interesting, and have the potential to change a lot of landscapes that today are relatively isolated.  Let’s consider what the forces of centralization are.

    First, there is a huge migration of software underway to the cloud.  In other words, software that is never installed on your machine or in your company’s datacenter.  It resides in the cloud and comes to you via the browser.  Examples include SaaS on the business side and the vast armada of consumer Web 2.0 products such as Facebook.  No category is safe from this trend, not even traditional bastions as should be clear from the growing crop of Microsoft Office competitors that reside in the cloud.

    Second, this migration leads to centralization.  The mere act of building around a cloud architecture, even if it is a private cloud in your own company’s datacenter, leads to centralization.  After all, software is moving off your desktop and into that datacenter.  When many companies are aggregated into a single datacenter, into a SaaS multi-tenant architecture, for example, further centralization occurs.  When you offer a ubiquitous service to the masses, as is the case with something like Google, the requirements to deliver that can lead to some of the largest datacenter operations in the land. 

    Third, there are the afore-mentioned economies of scale.  Google has grown so large that it now builds its own special-purpose switches and servers to enable it to grow more cheaply.  The big web empires are all built on the notion of scaling out rather than scaling up, and they run on commodity hardware.  Because they have so many servers, automating their care and feeding has been baked into their DNA.  Not so with most corporate datacenters that are just beginning to see the fruits of crude generic technologies like virtualization that seek to be all things to all people.  Virtualization is a great next step for them, but there are bigger steps ahead yet that will further reduce costs.

    Fourth, the ultimate irony is that centralization begats centralization through network effects.  This is the story of the big consumer web properties.  Every person that joins a social network adds more value to the network than the prior person did.  The value of the network grows exponentially.  This connectedness is facilitated most easily in today’s world by centralization.  Vendors that start to get traction increase their network effects in various ways:  Amazon charges to bring data in and out of their cloud, but not to transfer between services within the cloud.

    Lastly, there are green considerations at work.  The biggest costs associated with datacenters these days are around electricity and cooling.  Microsoft is building a data center in Siberia, which is both cold and pretty central to Asia.  Consider this:  given the speed of light over a fiber connection, what is the cost of latency in having a data center somewhere far north (and cold) in Canada like Winnipeg versus far south (and hot) like Austin, Texas?  It’s 1349 miles, which, as the photon travels (186,000 miles per second) is about 7.2 milliseconds.  The world’s fastest hard drive, the nifty Mtron solid state disks I’m now coveting thanks to Engadget and Kevin Burton, can only write a paltry 80K or so bytes in that time:  not even enough for one photo at decent resolution.  So consider a ring of datacenter clusters built in colder regions.  Centralized computing is up north where the cold that computers like is nearly free for the asking: just open a window many days.  Or come closer.  Put it up on a mountain peak.  Immerse it near a hydro dam and get the juice cheaper too.  It doesn’t matter.  Laying fiber is pretty cheap compared to paying the energy bills.

    The next question is trickier: how do these clouds compete?  Eventually, they will become commoditized, and they will compete on price, but we are a long ways from that point.  At least 10 years or more.  Before that can happen, customers have to agree on what the essential feature sets are for this “product”.  I believe this is where software comes into play, and that should be a matter of great concern for the hosting providers of today whose expertise largely does not revolve around software as a way to add value.   As Eric Schmidt said (via Nick Carr) when he started saying Google would enter this market:

    For clouds to reach their potential, they should be nearly as easy to program and navigate as the Web. This, say analysts, should open up growing markets for cloud search and software tools—a natural business for Google and its competitors.

    Some will immediately react with, “Hold it a minute, what about the hardware?  What about the network?”  The best of the cloud architectures will commoditize those considerations away.  In fact, commoditization will start down at the bottom of the technology stack and work its way up.  The first stage of that, BTW, is already almost over.  That was the choice of CPU.  MIPS?  PowerPC?  SPARC?  No, Intel/AMD are the winners.  The others still exist (not all of them!), but they’ve peaked and are on their way down at various terminal velocities.  Their owners need to milk them for profit, but it would be a losing battle to invest there.  Even Macs now carry Intel inside, and Sun now carries the ticker symbol “JAVA”, a not-so-subtle hat tip to the importance of software.

    Hardware boxes are largely a dead issue too.  There is too little opportunity to differentiate for very long and the cpu’s dictate an awful lot of what must be done.  Dell is an assembler and marketer of the lowest cost components delivered just in time lest they devalue in inventory.  Sun still pushes package design, and it may have some relevance to centralization, but this will be commoditized because of centralization.

    Next up will be the operating system.  Again, we’re pretty far down the path of Linux.  Corporations still carry a lot of other things inside their firewalls, but the clouds will be populated almost exclusively with Linux, and we could already see that has happened if we could get reliable statistics on it.  Linux defines the base minimum of what a cloud offering has to provide:  utility computing instances running Linux.  This is exactly what Amazon’s EC2 offers.

    What else does the cloud need?  Reliable archival storage.  Again, Amazon offers this with S3.  Cloud consumers are adopting it in droves because it makes sense.  It’s a better deal than a raw disk array because it adds value versus that disk array for archival storage.  The value is in the form of resiliency and backup.  Put the data on S3 and forget about those problems.  This begins the commoditization of storage.  Is it any wonder that EMC bought VMWare and that a software offering is now most of their market cap?  Hardware guys, put on your thinking caps, this will get much worse.  What software assets do you bring to the table.

    3Tera is a service I’ve talked about before that has a very similar offering available from multiple hosting partners of theirs.  They create a virtual SAN that you can backup and mirror at the click of a mouse.  They let you configure Linux instances to your heart’s content.  Others will follow.  IBM’s Blue Cloud offers much the same.  This collection is today’s blueprint for what the Cloud offers in terms of a platform.

    But, this platform is a moving target, and it will keep moving up the stack.  Amazon just announced another rung up with SimpleDB.  For most software that goes into the Cloud, once you have an OS and a file system, the next thing you want to see is a database.  Certainly when I attended Amazon Startup Project, the availability of a robust database solution was the number one thing folks wanted to see Amazon bring out.  The GM of EC2 promised me that this was on the way and that there would be several announcements before the end of the year.  First we saw the availability of EC2 instances that had more memory, disk, and cpu, so that they’d make better database hosts.  SimpleDB is much more ambitious.  It’s a replacement for the conventional database as embodied in products like mySQL and Oracle that was designed from the ground up to live in a cloud computing world.  At one stroke it solves a lot of very interesting problems that used to challenge would-be EC2 users around the database.

    Along the lines of my list of factors that drive data center centralization, Phil Windley says the economics are impossible to stop.  Scoble asks whether MySQL, Oracle, and SQL Server are dead:

    Since Amazon brought out its S3 storage service, I’ve seen many many startups give up data centers altogether.

    Tell me why the same thing won’t happen here.

    There is no doubt in my mind that all startups will give up having datacenters altogether before this ends.  However, before we get too head up in assuming that SimpleDB gives us that opportunity, let’s drop back and consider what it’s limitations are:

    – It is similar to a relational database, but there are significant differences.  Code will have to be reworked to run there, even if it doesn’t run afoul of the other issues.

    – Latency is a problem when your database is in another datacenter from the rest of your code.  Don MacAskill brings this one up, and all I can say is that this is another network effect that leads to more centralization.  If you like Simple DB, it’s another reason to bring all of your code inside Amazon’s cloud.

    – All fields are strings, and they are limited to 1024 characters.  Savvy developers can use the 1024 characters to find unlimited size files on S3, as well as other methods like combining fields to get around this limit.  Mind you, a lot can be done with that, but it is again a difference from traditional RDMS systems and it means more work for developers that must overcome the limitation.

    – There are no joins, if you want them (and many proponents of hugely scalable sites view joins as evil), you have to roll your own. 

    – Transactions and consistency are also absent.  Reads are not guaranteed to be fully up to date with writes.

    – There is no indexing and a whole host of other trappings that database afficionados have gotten comfortable with.

    Mind you, serious web software is created within these limitations including some at Amazon itself.  In exchange for living with them, you get massively scalable database access at good performance and very cheaply.  And, as Techcrunch says, you may be able to get rid of one of the highest cost IT operations jobs around, database administration, and your costs are even lower.  Remember my analysis that shows SaaS vendors need to achieve 16:1 operations cost advantages over conventional software and you can see this is a big step in that direction already.

    There is no doubt that cloud computing will be massively disruptive, and that Amazon are well on their way in the race to plant their flag at the top of the mountain.  The pace of progress for Amazon Web Services has been blistering this year, and much more hype free than what we’ve gotten from the likes of Google and Facebook when it comes to platform speak.  It’s almost odd that we haven’t heard more from these other players, and especially from the likes of Google.  GigaOm says that Simple DB completes the Amazon Web Services Trifecta.  They go on to say that Amazon’s announcements have the feel of a well thought out long term strategy, while Google’s make it sound like the ad hoc grab bag of tools.  I think that’s true, and perhaps reflective of Google’s culture, which is hugely decentralized to the point of giving developers 20% free time to work on projects of their choosing.  The problem is that such a culture can more easily give us a grab bag of applications, as Google has, than it can provide a well-designed platform, as Amazon has.  Or, as Mathew Ingram puts it, while everyone else was talking about it, Amazon went ahead and did it.

    I’ve talked to a dozen or so startups that are eagerly working with the Amazon Web Services and having great success, as well as some frustrations.  They require rethinking the old ways.  Integrity issues are particularly different in this brave new world, as are issues of latency.  That matters to how a lot of folks think about their applications.  Because of the learning curve, I don’t plan to go out and short Oracle immediately, but the sand has started running in the hourglass.  There will be more layers added to the cloud, and over time it will become harder and harder to ignore.  There will be economic advantage to those who embrace the new ways, and penalties for those who don’t.  This is a bet-your-business drama that’s unfolding, make no mistake.  At the very least, you need to get yourself educated about what these kinds of services offer and what they mean for application architecture.

    Business located low in the stack I’ve mentioned will be hit hard if they don’t have a strategy to embrace and win a piece of the cloud computing New Deal.  We’re talking hardware manufacturers like Sun, Dell, IBM, and HP.  Software infrastructure comes next.  Applications that depend on low cost delivery, aka SaaS, are also very much in the crosshairs, although probably at a slightly later date.

    Welcome to the brave new world of utility cloud computing.  Long live the server, the server is dead!

    Related Articles

    Amazon Raises the Cloud Platform Bar Again With DevPay

    Coté’s Excellent Description of the Microsoft Web Rift :  Nice post on cloud computing at Microsoft

    Posted in amazon, data center, ec2, grid, platforms, saas, Web 2.0 | 10 Comments »

    A Little More Google Knols

    Posted by Bob Warfield on December 15, 2007

    Nick Carr confirms what I’ve asserted in my post on Knols:  Wikipedia frequently comes high in the list of results for a lot of things.  He asserts this is causing Google to move with Knol lest they be “the front door for a vast ad-less info-moshpit outfitted with open source search tools.”  I have another thought.  Consider all the painful machinations the SEO crowd will go through to get a decent ranking for some page you probably didn’t like when you got there anyway.  Now consider what creating high quality content ala a Wikipedia article does for your SEO.  Content is still king.  If you’re not getting the ranking you want, maybe you’re not producing the content the rest of the world wants.

    Robert Scoble says Nols sounds suspiciously like Mahalo.  It isn’t really if you look at what the two provide.  Google may deliver Mahalo’s claimed benefits (human input enhancing search) by using information gleaned from Nols in the way I mention, but other than that, I don’t see the two as similar, anymore than people compare Mahalo to Wikipedia very much.

    Posted in Marketing, Web 2.0 | Leave a Comment »

    « Previous Entries