SmoothSpan Blog

For Executives, Entrepreneurs, and other Digerati who need to know about SaaS and Web 2.0.

Archive for April, 2009

Innovation is Dead, Software Margins are Dead, Woe is Us

Posted by Bob Warfield on April 30, 2009

I’m watching with some amusement a back and forth among some blogging colleagues about the demise of software company margins and all the terrible things that will come of that.  It’s the old story where the bloom is off the rose, we’ve lost our youth and innocence, and it just isn’t fun any more.

To that I say, “Hogwash!”

Don’t confuse the natural process of breaking down and cleaning up the carrion of the last generation with the (un)likelihood there will be another generation.  This cycle has been going on forever.  We didn’t get very far along in the Client/Server generation before nobody remembered the mainframe software guy’s names any more.  It hasn’t been that long since we were all reading PC Magazine and Computerworld to find out what was happening in the world (let alone the various newspapers that were falling like flies), but in a little while longer, we won’t remember their names either.  At some point in the SaaS generation, the same will be true for the on-prem gang, and anyone it isn’t true for will have made the transition to SaaS.  Yeah, I know we have a lot of skeptics, but it’ll happen and is happening as we speak. 
There is still tremendous wealth creation going on in software, and there are high margin businesses to be had.  There is still a lot of innovation left to be done.  There’s plenty of capital despite the handwringing over VC returns (which I ought to post about as well).  The Internet reduces friction to the point where innovators can get big extremely quickly and with relatively little capital. 
Liquidity ain’t what it used to be post-SOX, but it just hasn’t been that long since Google and it’s peers were created to declare the game is over.  Or rather, please declare game over, because that provides the necessary closure to get the next game started.
How many times have we heard it?
Motorola owns the cell business.  Oops, Nokia owns it.  Oops, iPhone is killing them.
Computer guys don’t get consumer electronics and will never compete with the likes of Sony.  Just look at that Walkman business!  Oops, here comes the iPod.
Microsoft has locked up the desktop.  Oops, I still use their apps, but that isn’t where I spend the majority of my time on my computer.
And on and on.  Always the declaration that we’re done, but we’re not.  Always the analogies to cars, planes, and probably at some point trains (“I think there is a world market for maybe five computers” would be a quote contemporary with trains in 1943, allegedly by Thomas J. Watson).  But computers are far more versatile machines than those engines of transportation ever were, and we haven’t tapped even 5% of their potential yet.  Even the most brilliant are subject to assuming the computer’s limits are in sight (“640K ought to be enough for anybody,” supposedly said Bill Gates, though he denies it).  Heck my 5% estimate is probably wildly wrong too and we have even further to go.
Sitting at the bottom of an economic downturn that’s about to head up is not the place to declare you have the vision to see the future and that future is limited.  There are no limits because there is no spoon!
Related Articles
Dennis Howlett is enamored with Open Source and SugarCRM.  He says it signals lower margins forever.  I still disagree, but Dennis is always worth the read.
Vinnie Mirchandani writes of “those who forget history…”, another excellent perspective.

Posted in saas | 8 Comments »

iPhones Not Used For Work? Balooney!

Posted by Bob Warfield on April 28, 2009

Sometimes folks struggle too hard to extract a headline from data when they should be trying to extract some new meaning.  So it is with Compete’s recent proclamation that the iPhone is not used for work.  Nothing could be more silly, as anyone that owns an iPhone will happily tell you.

The root of the problem is a simple one:  if you own one of the “other” SmartPhones, it isn’t good for anything but work.  In fact, it isn’t even that good for work.  It’s good for email and that’s about it.

The iPhone, by contrast, is a universal handheld web browser that can be augmented with zillions of apps.  Yes, I can and do check my email.  Yes, I can and do check Twitter.  Yes, I can and do read blog posts via the mobile edition of Google Reader.  Yes, I can and do make phone calls as well.  Yes, I take pictures.  In fact I often take pictures of white boards at work that we’ve just finished scrawling our key strategic plans and architectures on.  Yes, I use the maps function coupled with the web browser (so I can double check the address) to make sure I’m in the right place for a meeting.  Trust me, I have been to some unlikely locations, only to have my iPhone tell me I need to walk 15 feet further down the street.  I did so, peered in the window, and saw the sign for the business I was looking for just around a corner.  No evidence of it on the street.  Thank you iPhone!  And yes, I look up all sorts of things in meetings by Googling on my iPhone.  This makes me much more productive in those meetings.

Do I use my iPhone for work?  Absolutely!  I use it more than I ever did the gaggle of Blackberries and Treos that sit in a drawer at home unused.  To say otherwise is just Compete busy linkbaiting to get ink for their service.

Does that mean I don’t use it for personal time?  Absolutely not!

The iPhone will change your life, at least the telephonic part of it.  Towards the end of last year my family drove down to San Diego to pick up our new Schnauzer puppy, Butch.  My iPhone was with us all along the way.  It helped us to find great places to eat, and it entertained the kids.  I upgraded to a 3G about four months ago, and my 15 year old son inherited my old but still working perfectly 1st gen iPhone.  He is never without it and is constantly showing me fun things to do on iPhones that I will never find the time to track down.

If asked, do I use my iPhone more for personal than work use?  Like Techcrunch writer Robin Wauters, I would probably fess up that I do, but only because I do everything I can on my iPhone.  The other phones are stuck in Web 0.5 (not even 1.0, let alone 2.0) land. 

Is it any wonder that Tim Cook says Apple doesn’t need a netbook because they suck?  Apple already has netbooks in a far more convenient form factor.  If you’re going to lug a laptop, make sure it does a whole lot more.

Posted in apple | Leave a Comment »

Change the Environment to Change the Outcome

Posted by Bob Warfield on April 27, 2009

Creativity +  Passion +  Drive + Initiative +  Fun = Excellence

These are all qualities most of us would like to associate with our businesses.  Yet all too often we’re so focused on the business results, we forget to feed these qualities.  Business results can be great fun, and can involve a lot of passion.  But they can also become an endless focus on mind numbing blocking and tackling.  The solution is not to ignore the blocking and tackling.  We can’t.  In fact, we have to do more of it and do it better than the other guys if we expect to win.  What we can do is get our heads up every now and again and focuse briefly on something to do with the positive attributes that lead to Excellence.

I live in the coastal village of Santa Cruz, California, which is quite a drive from my workplace in Mountain View.  People tell me I made the wrong life choice.  I spend so much time at work, I should be living in Mountain View and visiting Santa Cruz.  But that’s just not how I see it.  True, Santa Cruz is not my whole week, but I get to experience it every day.  It lifts my spirits and gives me a profound break from the work-a-day grind.  That break helps me to function better at work when I get back.

Why not get such breaks more frequently?  Why not make the work environment itself offer a boost to your cognitive energies every time you turn around?  I see this sort of thing as the cognitive equivalent of a palate cleansing.  If nothing else, I try to jump up from my desk and go have a high energy conversation with someone in the company as I finish each project, just to reset my perspective before I tackle the next project.

I’ve done some pretty zaney things in my own past to keep the juices flowing.  Back in my Borland days, I kept a 700 HP twin turbo engine I built (cars and hot rodding them is an after hours passion for me) in the middle of my office.  It was a personal reminder of things I had to get done for Borland before I could go play with it, but it also served as a zaney source of energy around the company.  Our CEO Philippe Kahn made it a mandatory stop on any tour he was giving to a visitor.

I read a wonderful post by Robert Scoble on Zappos culture.  This retailler is famous for fabulous customer service, but Scoble’s story of what it’s like to work there is even more interesting.  They have some powerful cognitive palate cleansers at work there!  Every aspect of the culture is in tune with the idea, as a matter of fact.  It’s no wonder they generate such great customer service.  How can you expect to deliver wonderful service unless you’re having a good time doing it yourself?

Posted in business, Marketing, strategy | Leave a Comment »

Pick Anything Favors SaaS, the Cloud, and Community

Posted by Bob Warfield on April 24, 2009

Loved Seth Godin’s post on the calculus of change.  In it, he talks about how a change in operating systems (from DOS to Windows) opened the door for people to reconsider their choice of word processors.  The incumbent top of the heap word processor of the day was Word Perfect.  But when Windows came out, Word Perfect wasn’t ready and Word was. 

The point of maximum disruptive timing is the point where people have to pick something.  Pick anything. 

Are we at that point today?

I’ll argue we are for a whole host of things.  Old systems are showing their age.  The last major revolution in my company’s (Helpstream’s) Customer Service world was 10 – 20 years ago.  That means it’s time to retool.  The same can be said for most Enterprise Software.  We live in an age of Java, MySQL, Open Source, Cloud Computing, and Web 2.0 Social Media.  None of that existed when the current set of incumbent top of the heap Enterprise Vendors made their bones.  Each one involves powerful disruptive forces. 

They are so powerful that they even affect demographics.  People that grew up with all that want it.  It has changed how they live their lives, how they interact, and what tools they insist on having to hand.

Having to retool, whether because the economy forces another round of efficiency improvement, or your competitors have already retooled and reaped the benefits, or your next generation of employees won’t use your status quo creates that “pick anything” environment that Godin so eloquently says is the key moment for an insurgent, and the best time for a marketer to grow their audience.

Note that the incumbents and most of the players still in place who chose them will not see this.  But they will feel its impact later, when they realize they have been late to the party and lost out as a result.

Posted in cloud, Marketing, saas, Web 2.0 | 1 Comment »

A Vision for Oracle’s Cloud Platform: The Red Cloud

Posted by Bob Warfield on April 22, 2009

Helpstream’s CEO (my boss), Tony Nemelka, penned a great piece on the Helpstream Blog about what the Oracle/Sun acquisition might mean.   A lot of attention has been focused on the potential for negative outcomes.  Will Oracle kill MySQL or at least damage it with worse than faint praise (fascinating post by fellow Enterprise Irregular Josh Greenbaum)?  Other note (quite rightly) that Oracle can’t really kill MySQL because of its Open Source basis.  I found WordPress founder Matt Mullenweg’s post to be particularly eloquent of this.

What I found intriguing about Tony’s post was the more positive scenario it envisions.  Read the post, but let me summarize for purposes of this discussion.

Tony was recently in Japan and has a long history there having been an executive for PeopleSoft, Epiphany, and then Adobe in charge of the region.  Needless to say, his contacts are pretty high up the food chain, so they know what’s going on.  In that world, the System Integrators are the gatekeepers for the market.  They’re very powerful, and the interesting discovery Tony made is that they absolutely love  It’s not hard to see why.  The SaaS model squeezes the SI ecosystem.  The normal meat and potatoes business around just getting on-premises software installed is greatly reduced.  The business of just keeping the lights on is almost non-existant for SaaS.  Yet SI’s have a lot to bring to the table.  A good SI often understands the Domain, its Best Practices, and the key Business Processes better even than the software vendor.  Having access to a SaaS platform makes it possible for the SI to turn that valuable knowledge into product which can then be sold.  That’s why having a platform on which to do that is so important to them.

Tony goes on to speculate that Oracle is picking up the components necessary to create such a platform.  If nothing else, Oracle’s Japanese SI’s are screaming that they need one.  I have to imagine SI’s everywhere are grokking the essential value of a platform to the SaaS ecosystem.  There’s nothing about the Japanese market that would make that a unique requirement there.  So far, Oracle is really stuck in that department.  I suppose they would argue the coming Fusion represents such a platform.  At the same time Sun, like every big hardware vendor, was hard at work crafting a Cloud strategy.  They all know the Cloud is a tidal wave that can profoundly impact their businesses.  The Cloud represents the federation of many smaller deals into fewer gargantuan deals.  One over simplified way to view it is as a whole new sales and marketing channel.  Failing to suit up for the game guarantees a loss and the stakes are high. 

Some I’ve talked to say that Oracle just “doesn’t get it.”  They don’t believe in SaaS, they don’t understand SaaS, and they can never execute this kind of nuanced strategy until it is way too late. 

The idea that Oracle wouldn’t try or wouldn’t deliver anything is not something I’d want to put too much money on.  It’ll probably take them longer than anyone would like, or they might surprise us too.  Even Oracle can’t really fight the Cloud/SaaS tidal wave.  Remember that the guy at the top believes it in, don’t forget that Larry Ellison has put his money into SaaS companies many times.  It happens to be inconvenient at the moment for the various financial metrics Wall Street cares about for Oracle to switch wholesale to SaaS, but even that is something they can manage over time.  Also, we know Sun was working hard on their own Cloud strategy.  Suddenly there is a lot more Cloud DNA coming into Oracle. 

I suspect Oracle’s vision of what a PaaS (Cloud Platform) is will be a lot different than what you or I might choose.   If nothing else, it won’t be a clean sheet of paper approach.  Let’s think about what it might be.

First, I would expect the initial version will not be multitenant.  Multitenancy as it is delivered today is too deep in the guts of the application.  It forces too much change to architectures, which creates too much adoption friction for a platform at the outset.  Oracle will want to deliver the promise of running any app on their Cloud Platform, or at least any app built to run on their now very comprehensive stack.  It spans hardware (SPARC et al) to OS (Solaris) to DB (Oracle/MySQL) to App Server (BEA) to Business Intelligence (Hyperion) to <you get the idea, phew!>. 

Second, if not multitenant, what?  Think virtualization.   Don’t they need to buy VMWare to get that?  They may buy VMWare, but they don’t need it for virtualization.  Sun Solaris has a wonderful virtualization capability built right in.  I’ve used it before to create a SaaS application and it works extremely well.   Imagine an Amazon-like capability to start up these virtual Solaris machines.  If Oracle is smart, and they usually are, you’ll be able to start up virtual “appliances” in the Red Cloud (my new name for Oracle’s Cloud) that deliver database, app server, and many other functions.  Consider:

–  Storage:  This one is obvious.  Sun has a big storage business, Jonathan Schwartz has blogged about their great ZFS storage technology, and so Oracle can easily deliver the “Amazon S3” storage piece of the Red Cloud.

–  Identity Management:  Control over who logs on to the Cloud.  Sun is strong in Identity Management, for example.  Oracle already had a business there.  The combination of the two would create a leader to rival IBM and possibly be the world leader.  Having Identity Management built into the Red Cloud would be a decided work savings over Amazon, for example.  And Identity Management is one of those things on-prem apps are used to farming out to another module.  Hence it would facilitate their migration to the Red Cloud.

–  Business Intelligence:  This is another of those modules everyone wants to OEM instead of having to build for themselves.  It’s an ecosystem component, like Identity Management, ETL, and a host of other things.  Oracle can again deliver a virtual appliance in the Red Cloud that makes it simple to connect.

–  Integration:  This will be essential to making the Red Cloud Appliances work.  But when you have one vendor that controls so much of the stack, they should be able to make it work better than anyone else.  This is where the vision of being the “Apple for the Enterprise” can best be seen.  This is where a lot of the Fusion work, as well as work from BEA could tie in.

OK, that’s a pretty darned impressive first tier vision if you ask me.  It’s taking the old pitch about buying a suite from one vendor and ratcheting it up significantly in terms of scope.  Not everyone will buy into it, but Big IT might just need something like this before they can start moving to the Cloud.   If Oracle can deliver such a thing, it will be an enormous business.  Someone I was talking to recently said they thought Microsoft was Oracle’s ultimate acquisition target, but I think Larry has a shot at rivaling much bigger entities if he can execute.  The IBM’s and HP’s are starting to appear on his horizon.

But there are some warts.  It will be a hodgepodge unless some Fusion-like glue can make it all fit.  It will be an enormously complex offering to market and sell.  It will be nearly impossible to take in the scope of it or understand all of it.  All of the Enterprise complexity that Big IT loves, but that SaaS has tried to ameliorate will be there.  Granted, this is better than the old on-prem complexity.  Oracle can deliver at least some of the SaaS advantages.  But what about cost?  Can we really get the SaaS cost advantages without multitenanacy?  In a word, “No!”  I just wrote an article about the pitfalls of thinking virtualization is a substitute for true multitenancy.  I stand by every word I said.  But Oracle has a unique opportunity to virtualize multitenancy itself, not in the first iteration of the Red Cloud, but in later iterations.

Whoa!  What the heck is Bob on about now?  Virtualizing multitenancy, what does he mean?

What I mean is a mechanism whereby single tenant applications can be made to have all the benefits of multitenancy without radical architectural change.  There are two ways this can happen. 

The approach is to have the database itself virtualize multitenancy.  The most popular model for multitenancy is what I call columnar:  a column in the database tells which tenant each record belongs to.  A suitable feature set in the database server can completely automate this and make it radically simpler for the application to work along this model.  A second common model gives every tenant their own set of tables.  Here again, special support in the database can radically simplify the implementation.  Note that Oracle already does partitioning (they may have invented it, but I am not sure of that), which is a feature that makes a bunch of physical tables look like a single table and that greatly improves scalability.  So now Oracle could deliver a Multitenant DB Appliance in the Red Cloud.

The second approach attacks the costs of being single tenant.  In my post on the subject, I talked about the idea of fixed costs and variable costs.  The difficulty is that the database server uses up a bunch of system resources on each virtual machine even if there is no data loaded into it.  This is because the database server is unaware of the other database servers in their respective virtual machines.  They are isolated.  But what if they were aware of each other and could pool those fixed cost resources to reduce the overhead?  What if the operating system itself facilitated this?  Those fixed costs could be dramatically reduced.  Moreover, if a comprehensive integrated feature set was aimed at reducing the cost of administering such as system, we would likely start closing in on true multitenant efficiency levels. 

That would be my vision for Oracle’s Red Cloud.  It would be a first-class Cloud platform, although much more Enterprisey and Old School than the New Cloud Age SaaS and Amazon-style Cloud visions.  Oracle and BigIT will view that as an advantage.  The biggest challenge in all this will be execution.  It is a gargantuan task on a level Oracle has never delivered before.  It involves both coordinating a lot of existing parts and pieces as well as delivering some genuine innovation (Virtual Multitenancy will be non-trivial!).   They may not be able to pull it off.  But the stakes are very high, and they will have years to work at it.

What do you think of the Red Cloud?

Posted in amazon, business, cloud, data center, saas, strategy | 4 Comments »

Give Your Sales Pipeline an MRI Scan

Posted by Bob Warfield on April 21, 2009

I had a chance to catch up with Lucid Era’s Ken Rudin, VP of Market Development and Co-Founder, last week.  It’s been a little while since I last talked to Ken, so there was a fair bit of news. 

Lucid Era has always been high on my list for having been innovative in thinking about how to take as much friction as possible out of the SaaS buying cycle.  They have a concept called the “Pipeline Health Check”  that is very slick.  You give them access to your CRM system, they process the data through their analytics solution, and 48 hours they’ll tell you meaningful things about what’s going on.  Hence I liken it to an MRI scan for your Sales Pipeline.  But there’s more.  LucidEra has decided to sweeten the pot.  If they don’t find at least $100,000 of change (up or down), they’ll pay you $1000.  Considering the Health Check is free, that’s an interesting guarantee.

There are some other announcements associated with the latest release of LucidEra, including some new dashboards built with Flex technology as well as more historical trending and analysis.  One of the coolest features is for “guided analysis.”  Using analytics tools effectively is one of those things that requires talent.  You either get it or you don’t.  One of the hardest things to “get” is what question to ask next based on the analysis your looking at right now.  LucidEra’s guided analysis helps solve that problem by telling you what other reports other folks looking at the one you’re seeing found useful.

LucidEra is built on a full analytics platform, so there are general purpose tools underneath.  Where they really shine is in turning those tools and platform into solutions to real problems that don’t take a Wall Street Quant to decipher.  Check them out!

Posted in saas | 3 Comments »

Oracle Buys MySQL, Java, and some other stuff. Now what?

Posted by Bob Warfield on April 20, 2009

Big headline this morning with Oracle buying Sun after IBM had walked away.  The deal is valued at $7.4 billion, which is just a hair over a 40% premium.  Not too bad.  Pretty typical in fact for an Oracle acquisition.  After a close call, Sun has avoided being Yahoo 2.0 in the mergers and acquisitions game.  Congratulations.  Now what?

I wrote my headline as I did for a reason.  Most people think of Sun as a hardware company, but there’s more too it than that, and hardware is the least of what’s important for this deal.  First, consider the perspective on deal size.  This is not the largest acquisition Oracle has made.  It’s clearly still a strategic deal, but there is room given the deal size for Oracle’s interest to be something different than hardware.  That’s not what Oracle is saying, but it is a practical possibility.  With that perspective in mind, let’s consider what the deal might mean:

–  There will be wholesale carnage at Sun.  The company had not been especially healthy before the acquisition, but Oracle is pledging to deliver a 15% margin in the first year.  I don’t doubt they can achieve that number, but doing so will require a vigorous shake up.  Costs will be cut.  Premiums will be tacked on in strategic places.

–  Oracle has consolidated its hold on key IT technologies relatively cheaply.  MySQL, Java, and Solaris/SPARC are all de facto standards in the marketplace.  Java is particularly compatible with Oracle’s BEA unit.

–  Startups and web companies should be wondering how much longer the free lunch is going to last.  Oracle has never been known for its generosity, and an extremely large market has been dining for free on MySQL and Java for years.  Will Oracle change that dynamic in some way?

–  Big Companies take note:  prices will be going up and service levels + innovation down.  Mostly this doesn’t phase you, I know, but perhaps this would be a good time to rethink that whole “fewer bigger vendors” thing?  Start loading up on SaaS solutions and let those vendors deal with it. 

–  Assuming Oracle does something to make one or the other unpalatable, would you rather have to move off of MySQL or Java?  Personally, I’d rather deal with moving off MySQL.  There are several viable alternatives in the market that would involve a pretty minimal porting project and that are known to perform well, some even think better than MySQL.  Porting from Java to some other language is problematic to say the least, and nigh unto impossible to be realistic.  Vinnie Mirchandani has labeled MySQL “MyToast”, and says Oracle can’t possibly let it continue to compete with it’s mainstream server.  Larry Dignan agrees, and I confess I am concerned  too.  Om Malik wonders whether Oracle will take care to keep the MySQL developers happy.  That presumes Oracle wants them happy, or hard at work improving MySQL further.  At the very least I would expect Oracle to be a lot less squeamish than Sun about setting up a tiered set of versions where only the bottom tier is really free.

–  You have to wonder what a further consolidation of the database world means for the likes of Microsoft’s SQL Server and IBM’s DB2.  If I were in their shoes, I would be gearing up the legal machinery to force Oracle to divest MySQL before they even get started for anti-trust reasons.  Don’t be surprised if some such comes to pass before it’s all over.  I think that would be a good thing for the industry, if only to save MySQL from becoming MyToast.

–  What about the app server world?  It doesn’t get talked about a lot, but most of the companies I talk to had moved off WebLogic and WebSphere to favor the open source alternatives like JBoss and Tomcat.  Can Oracle do something to Java to make it favor WebLogic?  I haven’t thought that one through, but if it’s possible, they’ll probably look into it.

–  And what about other vendors?  Mirchandani says John Chambers is probably chuckling.  Probably true.  IBM will have told themselves by now this is the best that could have happened and they’re glad they dodged a bullet.  At the same time, there are some butterflies deep down in their gut.  There is now a new combined hardware and software player that is of a scale to cause IBM some problems.  What to do?  Buy SAP would be my suggestion, but that’s the subject for another blog post to really dig into it.  And what of HP?  Well again, they’re probably with IBM, having convinced themselves Oracle will fumble the hardware ball and it won’t matter.  But they’ve got to be wondering what they should acquire in response.

–  Last crazy thought.  Oracle has poo-poohed SaaS for a long time.  Is there a scenario that rationalizes a Sun acquisition with a strong move into SaaS for Oracle?  I think there is.  First, vertical integration from apps all the way down to the hardware can be an effective recipe for lowering costs, although it does tend to slow innovation.  Cloud computing is all about minimizing costs, and there have been a number of interesting developments in terms of hardware built for the Cloud.  Sun also has Cloud initiatives underway.  The thing about the Cloud is it causes fewer larger transactions.  It bundles customers into gigantic data centers.  There is a lot riding on what these data centers standardize on.  Oracle could milk the external sales of SPARC to defray costs while their true strategic intent is to go big in the Cloud.  Big enough to own it.  Think about what a cost advantage Oracle could have in the SaaS world:  they don’t pay for the DB, they get hardware at cost, they don’t pay for the OS, they don’t pay for the app server, yada, yada.  Meanwhile, they can squeeze the margins of other SaaS companies who do have to pay for all of that.

These guys play for keeps.

Related Articles

A Vision for Oracle’s Cloud Platform:  The Red Cloud

Posted in business, strategy | 12 Comments »

Does the Cloud make Single-Tenancy OK for SaaS?

Posted by Bob Warfield on April 15, 2009

Multi-tenancy and all its flavors seems to be on people’s minds lately.  I just finished going back and forth with Phil Wainewright on some of the nuances of multi-tenancy and how it impacts the cost model for SaaS.  Phil’s post was on some of the sophisticated nuances of multi-tenancy as expressed by some of the latest announcements from the SaaS Vatican,

More recently, it has come to my attention both through my fellow bloggers of the Enterprise Irregulars as well as via a trackback to my blog post that there are those who are now saying the Cloud has made the world safe for SaaS vendors to forget multi-tenancy and plunge ahead with single-tenancy.

Not so fast! 

Color me skeptical.  If you don’t have a multi-tenant architecture, you’re going to argue it isn’t necessary.  That doesn’t make it right.  Before we get too far along, let me define multi-tenancy:

Multi-tenancy is a software architecture that allows multiple tenants to be hosted on a single box (or cluster of boxes) just as easily and economically as a single tenant could be hosted on the same configuration.

The bloggers taking the position that single tenancy is good enough hoist a variety of flags in support of their position, but for me, it boils down to answering one simple question about your SaaS business and its customers:

Fundamentally, do you have an application that can successfully run multiple tenants on a single box or not? 

If a single box has enough horsepower to run multiple customers for your app, the argument for single-tenant is completely (pardon my near-pun) untenable.
Salesforce runs 55,000 customers on 1,000 commodity servers.  You just aren’t going to be able to do that with a single tenant architecture no matter how much virtualization you choose to run.  If nothing else, virtualization runs afoul of a fixed cost/variable cost phenomenon very quickly.  A lot of the basic system software allocates fixed overhead, whether we’re talking about your DB server, your app server, your web server, or whatever.  Virtualization does not share the resources required for the fixed overhead, only the the variable costs.  Multitenant shares the fixed overhead too.  Those variable costs put an upper limit on the number of tenants you can shove onto a single box, no matter how small the tenant’s needs may be.
The new articles maintain that the Cloud fixes all this through the magic of elasticity.  Really?  That’s hogwash.  The Cloud at best and if you really architected your app to take full advantage of elasticity may help a little bit.  But most of the problem is database, and elasticity and databases so far remain a very hard technical problem to solve.  Try dynamically varying your partitioning and/or federation scheme to really scale up and down in real time in the Cloud.  It’s hard enough to get apps to scale to arbitrary Enterprise needs at all.  Try doing that in real time so you get multitenant cost savings?  Good luck!
So if you can’t run an average of 55 customers on each of 1000 servers like Salesforce, how many can you run without multitenant?  3?  5?  10?  What does that do to your cost versus true multitenant?  What does that do to the overhead of maintaining the servers?  What does that do to your cost of delivering the service and to the resultant cost model you have to saddle you customers with?  What does that do to you competively if you’re up against a company that does have the true multitenant cost advantage?
One example on the cost subject that I am familiar with:  a lot of the Social Software companies wind up charging by page views or total participant seats.  In many ways, this is anathema to community where you should do everything you can to encourage participation and not penalize it.  This is especially true for outwardly facing communities where the company wants a predictable cost model and can’t imagine being charged by their continually changing customer base or especially the changing usage patterns of that base.
In fairness, there are some business model + company + market + architecture combinations where it wouldn’t matter because you can’t run multiples on a single box.  If you’re strictly selling to organizations that can’t run on a single box no matter what, single tenant is fine.  Perhaps this is what these other bloggers are saying, but I’m skeptical any Enterprise 2.0 app would have that requirement, and that’s the kind of software these bloggers are describing.  FWIW, Helpstream will run 150 customers and nearly 400K seats on a single box loaded to about 10% of capacity.  That’s nothing though.  Look to the Googles, Facebooks, Amazons, eBays, Twitters, and similar big web properties to see real capacity.  Now combine that kind of capacity with multiple tenants.  It is a powerful competitive position to be in.

As I say, one can imagine combinations where you couldn’t combine multiple tenants onto a single box.  A honking big transaction processing ERP app might be one.  For another, at Callidus, I had customers using as many as 150 CPU’s to generate all the sales commission calculations for a huge salesforce.  To give an idea, we had telcos paying 20K sales reps and insurance companies paying 250K reps.  That’s a lot of transactions paid to a lot of people!  But those kind of Enteprise deals are very unusual for SaaS companies, and that kind of app is pretty unusual too because of the number of transactions being processed and the complexity of the business logic.  Still, such apps could be successful in those markets with single tenancy.
The articles go on to talk about the advantages of being able to customize these multiple instances.  Frankly, that scares me too, because the whole SaaS model really starts to break apart there when you decide to radically customize each instance.  It may be a value add, but it is a radically different value add than SaaS.  In fact, at that point, it’s a hosted ASP model, not SaaS.  Useful for some organizations, but there is a reason that model never achieved broad market appeal. 
Lastly, let’s talk about the whole security business.  This is the 800lb Red Herring in the room.  The minute you go SaaS or Cloud, you have outsourced that problem.  You can listen to vendors argue all day long about which architecture is “safer”, but that is an over simplification of the myriad factors that matter to the point it is just marketing and not substance.  It has as much to do with process as code architecture, which is why most of the security related standards like SAS70 and HIPAA don’t spend a lot of time on software architecture so much as the processes that surround that software.  Don’t take my word for it.  Look at what happened to Amazon around the whole AmazonFail incident for lack of process on an area that didn’t even involve any code.  Their problem was due to a data change.

BTW, the multi-tenancy imperative gets stronger constantly due to the multicore crisis.  We no longer get faster cpus (i.e. faster clock speeds) every 18 months according to Moore’s Law.  Instead, we get twice as many cores.  The easiest way to take advantage of more cores?  You guessed it: stack more tenants into the same box. 

For more, read Michael Dunham’s excellent post over on Haut Tec.

Posted in cloud, enterprise software, saas | 12 Comments »

AmazonFail Shows Data Matters Too

Posted by Bob Warfield on April 14, 2009

No software company in their right mind would change code and move it to production without extensive testing to make sure the new code wasn’t broken.  It’s a tried and true business process supported by tools like automated build (gather up all the files, compile as necessary, package, and produce a running version), source code control systems (check in and check out with auditing so all the files that go into a particular version are known and verified), and so on.  That’s all fine and well, and though I do sometimes find a company that hasn’t made it to even that basic stage of operational process, there is a level of complacency associated with having such a process working.  The AmazonFail incident shows us that there is a lot more than just program code at stake here. 

First, what was the AmazonFail incident?  It seems that Amazon suddenly started delisting Gay and Lesbian publications from their sales rankings.  This had the effect of removing the books from search according to the WSJ.  Needless to say, the incident resulted in a great hue and cry since many assumed Amazon had done this intentionally and took it as evil behavior (along the lines of Google’s “do no evil” mandate).  Before long there were charges of  online censorship, and Twitter and the blogosphere were lit up bright talking about it.  Rumors even emerged that the incident was a result of hackers.  Eventually the whole thing became known as “AmazonFail“, and there is a hashcode for that on Twitter if you want to read all about it.  As I write this, “#amazonfail” is the third most popular search term on all of Twitter.

Clearly it’s been a major PR black eye for Amazon, but what caused it?

Amazon calls it, “an embarassing and ham-fisted cataloging error.”  Ultimately it affected over 57,000 titles including books well beyond the Gay and Lesbian themes first reported.  A more detailed internal story comes to us from Andrea James of SeattlePi:

Amazon managers found that an employee who happened to work in France had filled out a field incorrectly and more than 50,000 items got flipped over to be flagged as “adult,” the source said. (Technically, the flag for adult content was flipped from ‘false’ to ‘true.’)

“It’s no big policy change, just some field that’s been around forever filled out incorrectly,” the source said.

Amazon employees worked on the problem well past midnight, and then handed it over to an international team, he said.

Doesn’t this sound just like the sort of problem that can be caused by making a minor code change and then rolling out the changed code to production without adequate testing?  First, it was a minor change.  One field was filled out incorrectly by one person in France.  Second, it created a huge problem as minor changes in code often can.  Third, the problem was not immediately obvious, nor was the cause, so it got pretty far along before it could be fixed.  Yet it wasn’t code, it was just data.

In this day and age of Cloud Computing, SaaS, and web applications, data is becoming increasingly just as critical as code.  Metadata, for example, is the stuff of which customizations to multi-tenant architectures are made of.  In that sense, it is code of a sort.  “Soft” heuristics are common in search applications that have to decide which words to ignore completely, how to weight different words, which words might be profanity (or adult content in this case), which words are positive, negative, might be Spam related, and all the rest.  That’s all critical metadata to a search engine such as Amazon’s.  There’s a lot of other critical data to consider including credit card and other privacy-related information, financial information (what if someone gets the decimal point wrong when entering sales tax for a particular area?), and so on.

Data drives modern applications so completely, that we have to think about what this means for the security and robustness of our applications.  We’re still in our infancy on that front.  Modern software will test all the data entered to be sure it doesn’t contain a malicious payload (the SQL injection attack is one way to hack by entering special data in a field exposed to users) and there are many similar low level tests that are made.  But what about the business process that ensures the integrity of that data?  How can a single individual in France create such a big problem for Amazon by changing a little bit of data?

Let’s assume for the moment that we choose to treat that data as code.  That would mean we do some or all of the following:

–  Archive all the data changes in the equivalent of a source control (also called a “version control“) system.  We’d know every version of the data that was entered, who entered it, when they entered it, and there would be comments about why they entered it.

–  The incorporation of that data into a production environment would happen only at carefully orchestrated times (called “releases”) and only after testing had been done to ensure no problems were created.

–  If a problem manifested itself, an automated system would be available to roll back the changes one or more versions until the problem went away.  This is an important step with extremely serious problems because earlier releases will have been functioning long enough that there are no known serious problems.  Rolling back gives time to find the error without exposing all the customers to it in the meanwhile.  The error is eliminated and then the updated change is tested, and rolled out again.

Does that sound hopelessly painful as a process?  It’s exactly what most software developers go through for even the slightest change to code.  I’ll admit it would be too painful for every data change.  Amazon must add, delete, or change thousands of new listings every day.  Each one can’t be a full development release cycle.  But it does seem that applications should have some safety valves built into their fabric, and into the all-important data that they rely on.  Changing a listing is different than changing some data that almost 60,000 books rely on in search.  That data should be marked as sensitive and handled differently. 

There’s lots of architecture and process work to think about in order to avoid or minimize similar problems in the future for a whole host of applications.

Related Articles

It’s the Data Stupid.  Vinnie Mirchandani offers more examples of how we get into trouble with data.

TechCrunch gets it all wrong in a guest post by Marry Hodder.  Hodder argues unconvincingly that this was all due to an algorithm gone wrong and kept secret.  What happened is precisely NOT an algorithm.

If it had been an algorithm, it would have been an unambiguous set of rules by Hodder’s own Wikipedia-quoted definition.  Algorithms are explicit for their creators, and they are not accidental.  That they may be hidden from users does not detract from their explicitness, purposefulness, or unambiguity.

To assume an algorithm is at fault, is to misunderstand what an algorithm is, or to assume to Amazon purposefully set about creating an explicit and unambiguous set of steps to discriminate against Gay and Lesbian titles.  While the conspiracy theorists may like to natter on about this theme, it ain’t so.

The problem here, and my point in this blog post, is that we assumed we could focus on the algorithm and ignore the data that fed it.  Clearly a bad assumption.  In so doing, we allowed a minor change by an individual to create a major PR problem.  If we really think it was all a conspiracy against all things Gay, why not look for more explicit evidence.  Did Patricia Cornwell’s books about Kay Scarpetta, which include some Gay themes (Scarpetta is Gay) disappear as well?  I don’t think so.

It’s been true since we’ve had computers:  garbage in leads to garbage out.  The data is just as important as the algorithms.

Posted in amazon, cloud, QA, saas, software development | 2 Comments »

Can Corporate IT Operate as Efficiently as

Posted by Bob Warfield on April 10, 2009

Phil Wainewright penned an interesting analysis of what I had been calling’s “Green Crystals,” but more commonly known as multitenancy.  In the past I’ve said that the exact technology involved there was not really the important thing, it was just an interesting side story told for marketing purposes.  Put another way, there are lots of ways to skin the multitenancy cat, but that isn’t even the important cat to be after.

At one point in his post, Phil frames the discussion in a particularly interesting light:

So are’s green crystals just a marketing ploy, or is it really something that customers can’t easily replicate in their own private clouds, or by adopting rival cloud platforms?

I like this statement from a couple of angles.  First, it asks the legitimate question of whether the non-SaaS world can learn some interesting things from SaaS technology.  Second, it ultimately leads into a discussion of some technology around query optimization that Salesforce has patented, and whether those patents have given them an interesting monopoly now that will be denied others.

But before we delve into all that patented query optimization, let me make another controversial statement:

It’s just more green crystals and it is not the source of Salesforce’s advantage.

Why do I keep bringing up this silly green crystals notion?  It is human nature to want a reason to believe.  Marketing has trained us to look for the “silver bullet advantage.”  Green crystals marketing is a term I first heard coined at Borland years ago when I was VP Of Engineering.  Our products had loads of advantages over the competition, but marketing wanted one single reason they could tell customers why we had all those advantages, because its impossible to make a compelling sale off hundreds of small advantages.  The term “Green Crystals” comes from soap marketing.  Why is my soap better than yours?  Because it has Green Crystals, that’s why!

Likewise, SaaS has a myriad of advantages over other delivery vehicles, and one SaaS vendor may have a myriad of advantages over another.  But it helps to have Green Crystals.  In the past, it was enough just to be a SaaS vendor, but the new breed had to differentiate themselves from the old hosted ASP model.  Hence Marc Benioff brilliantly coined the multitenant argument.  Now there are getting to be a lot of SaaS vendors with multitenancy, so maybe it’s time to coin something new, “patented query optimization,” anyone?

Whether you believe they’re just Green Crystals or not, their only advantage to the customer is that they enable SaaS companies to deliver their service at a lower cost.  I have argued in the past that most of the savings is not due to multitenancy (or patented query optimization), but rather automation of the people costs.  I stand by that argument, but I want to drill down on it further with some real numbers.

Let’s start by considering the cost to deliver the service for a few SaaS companies.  Here, Salesforce really does have a tremendous advantage over most of its brethren.  Consider these numbers taken from the latest 10K filings for these SaaS companies:

–  Salesforce delivers $1 in revenue at a cost of about 12 cents to deliver the service.

–  Concur delivers $1 of revenue at a cost of about 32 cents–almost 3x what Salesforce spends.

–  Companies like Success Factors (about 35 cents) and Omniture (36 cents) have a similar much higher cost than Salesforce.

You can go analyze all the public SaaS companies, but the last time I did so, I don’t remember anyone doing better than Salesforce.  They know how to deliver their service cheaply!

Now the next thing to look into is what the cost components might be.  The point of multitenancy (and virtualization for that matter), is to share infrastructure as needed, amortizing it across a lot of customers, so that costs are saved through a reduction in waste.  All of Salesforce runs on only 1,000 machines.

That sounds really impressive.  Perhaps the competition is simply running 3x as many servers because of their inefficient multi-tenant architectures.  Is that the source of the cost differential?  Well, let’s start by figuring out what 1,000 machines cost.  I’m a Cloud guy, so I would buy them from Amazon.  I will assume that Salesforce can run their own data center at least as cheaply, so the cost of 1,000 servers on Amazon running 24x7x52 ought to be a conservative number.  It comes to about $4.7 million, which is less than 4% of Salesforce’s cost to deliver their service.    The competition would have to be using more than 5x as much hardware as Salesforce to account for that kind of cost differential from hardware alone.  Now you see why I have a hard time believing the Green Crystals are telling the whole story.

What else may be at work here?  It’s the cost of the operations people that drives the Lion’s Share of what it costs to deliver a service.  Consider a fairly low-paid ops person who is technical (DBA’s and the like) may have a loaded cost of perhaps $120K per year.  That cost is equivalent to 25 of those servers, BTW.   So how many ops people does the gap in the cost of the hardware versus the overall Salesforce service cost represent?  About 1,000!

Now I am not trying to say all of the cost differential is due to the cost of such ops people, but it does bear thinking about.  Certainly there are other costs involved too.   I have a friend running a SaaS company who privately says his Oracle licenses are the biggest cost after the personnel and ahead of hardware.  Most SaaS companies didn’t build on Open Source technology, so they are going to have all sorts of software license costs.  Still, I assert most of their cost is people.

Salesforce must therefore be radically more efficient at automating what it is taking other companies a lot of warm bodies to deliver.  So if Phil’s corporate clients want to even begin to approach the efficiency of a Salesforce (let alone another SaaS vendor), they have to focus on that kind of automation.

How good can it get?  That’s an interesting question.  Salesforce is using 1,000 servers to deliver a service to 1.5 million people, according to the Techcrunch article I linked to above.  Consider these numbers for Facebook that are taken from a great posting over at Diamond Notes:

–  Facebook at that time (early 2008) had 1800 MySQL servers, 10,000 web servers, and 800 Memcached servers.  They had a total of 12,600 servers, in other words.

–  We’re hearing Facebook has 200 million users today, let’s assume they had 100 million a year ago.  May have ben less, but we need to make a guess.  That means they’re housing nearly 8,000 users per server, versus Salesforce housing about 1,500.

–  Even more interesting are the figures on how many MySQL DBA’s each company in that survey uses.  Facebook has just 2 DBA’s keeping 1800 servers happy!  That’s a couple orders of magnitude better than anyone else in the survey.  Clearly Facebook has been very successful in automating a lot of operations work.

All of these factors will play a role.  However, given that a single operations person is equivalent to 25 servers in cost, and given that we have the evidence of companies like Facebook, massive automation of the manual tasks is the big cost reduction opportunity moreso than making more efficient use of the hardware.

Imagine being able to keep Salesforce’s 1,000 servers humming with just 10 or 20 ops people.  Can corporate IT ever get there?  Sounds very hard to me when you consider all the advantages a company like Salesforce has over Corporate IT.  If nothing else, they can change the software to facilitate the automation instead of having to go in after the fact with third party tools.

Posted in data center, saas | 5 Comments »

%d bloggers like this: