Amazon Web Services: The De Facto Cloud API?
Posted by Bob Warfield on July 12, 2010
Read a couple of posts last week that coalesced some thoughts I’d been having into this one. First was the fascinating rumor about a Google EC2 clone. Hat tip to High Scalability Blog for putting me on to this one. The second was James Urquhart’s musings about the desirability of the Amazon API’s as a standard.
James and most of his commenters are worried that we might standardize on something that doesn’t have all the bells and whistles. Guess what guys, standards NEVER have all the bells and whistles. By the time people get done arguing about them and they get enough momentum and use to be more than just standards in name, the world has moved on. So what? If we waited for standards to be perfect, we’d would have none. The point of the standards is that they’re good enough to help reduce friction in areas where innovation at every turn is no longer desirable.
Are we there yet with the Cloud?
I think so. I’m not saying there is no innovation left to do–there is tons of it coming, and it’ll all be good. But, there is value at this stage in having a standard too. There has been a lot of success demonstrated with the Amazon Cloud from companies large and small. I would be curious to hear whether Amazon thinks their API should be a standard (and whether they’re prepared to give up some control in exchange), or whether they think it still has too much growing up to do.
I look at it like this:
First, the wonderful thing about standards is there are so many to choose from. Just because we annoint Amazon as one such standard does not mean there can never be others that are completely different. It does not mean the standard can never have a superset or optional features to cover many of the cases Urquhart raises. And remember, there is still that period of making it into a standard during which it can be molded a bit before it settles in.
Second, speaking of supersets, I think it is important to think about Cloud Standards as a layered model. I would definitely not put all of Amazon’s offerings into the standing. Perhaps just EC2, EBS, and S3. Note that S3 is spreading even more rapidly than EC2 as a de facto standard. These three seem pretty benign, reasonably abstract (meaning they don’t expose too much ugly detail about what goes on underneath), and reasonably proven.
So what do we get for it?
If the standard works and matters, we get a lot more vendors supporting the standard. That’s a good thing, and something I have to believe every Amazon Web Services client would very much like to see happen. The second thing we get is that some, but not all, of the innovators will quit trying to reinvent the wheel that is the Amazon layer (IaaS if you must use an ‘aaS name for it) and they will move on to other layers. Depending on how much more innovation you think that layer needs, this is a good thing. For those that think it is a bad thing, there is still time for someone with a better mousetrap to put it out there and show us. But what if making it a standard causes a ton of innovation around how to solve some of the problems like I/O bandwidth behind the scenes without affecting the API? Wouldn’t that be an excellent thing? Heck, I’d love to be able to add an “11” to the AWS I/O speed control, who wouldn’t?
But time is a wasting. For whatever reasons, it seems like the other players are very late to the party relative to Amazon. If they wait too long, Amazon gets a de facto standard before the market has much leverage to pry loose some control. This is arguably what happened around the IBM PC, DOS, and Windows. BTW, you read that like its a bad thing, but it wasn’t. We accomplished a heck of a lot by not having to continue innovating over the bus, the BIOS, and so on. It just could’ve been a lot better if there hadn’t been so much control vested in Microsoft and IBM. As I write this, I wonder a little bit if players like Google really aren’t taking too long. Maybe AppEngine is the exact image of what they think the Cloud should look like, and the problem is the world just didn’t bite the way they have with AWS. Google could be saying, “Okay, we hear you, so we’ll do it your way now.” There has to be some reason they’d endorse S3 and potentially EC2.
If Google’s ready to go there, why not the rest of us?
cloudsigma said
I read your latest post with interest. Some of your points I agree with although some of them I am definitely at odds with!
I think the idea that IaaS can be defined by one common standard that allows any meaningful level of interoperability is not true. Fundamentally computing resources can be organised in different incompatible ways and those different solutions can be optimal for different users. It does mean the notion of a standard IaaS solution is false in my opinion. The current largest player, Amazon, has key limitations which, if adopted by the whole industry, would severely limit the applicability of IaaS as a product to many sectors in particular the enterprise sector. Lack of control of the software layer, lack of flexibility in networking, lack of control of load balancing, lack of root access, lack of flexible resource allocation, non-transparent pricing, non-transparent performance, lack of privacy (vendor access to server instances) etc. Most enterprises will not accept the loss of control and security that moving to a public cloud aka AWS/EC2 entails. It isn’t therefore surprising that we see a steady drumbeat of ‘concerns’ surrounding cloud computing that relate more to do with vendor implementation than fundamental limitations.
Adopting a standard at this stage of the technology curve would mean placing a speed limiter on innovation in the cloud space with regard to IaaS in particular. It isn’t even clear that adoption of the cloud is at all limited by a lack of any standard API. In fact, having specialised clouds tailored to the differing needs of difference market segments makes a lot more sense than a one size fits all strategy that is likely to please no-one. API integration is in fact a relatively minor part of the cost of moving clouds, far more important is the openness and use of proprietary architecture (largely arbitrary) that the vendor has employed.
In reality customers will likely look to use two or three vendors who offer value in the full meaning of the word i.e. in terms of many factors such as performance, cost, reliability, availability, scalability, controllability etc. That trade-off will have different solutions for different customers depending on their preferences and needs. Just as there isn’t one standard in traditional web hosting, there is unlikely to be a standard in cloud computing in the medium term. With all new sectors there is a clamour for an over-arching standard; the reality is this is rarely the long-term equilibrium position.
Customers are perfectly capable of discerning the real value factors when choosing a cloud provider without the need for a ‘standard’ for the sector. That means evaluating many factors of which the API is just one. In reality the real friction results for orchestrated vendor lock-in; reinvented tried and tested behaviour in a shiny new wrapper (IP addresses are a classic example of this amongst some IaaS cloud vendors).
As a founder of CloudSigma, I am fully committed to providing customers full control, in terms of networking, the software they install, access to data (full FTP access in and out), transparent billing, unbundled utility computing style resources and more. All are feasible and deliverable today in a high performing, high availability environment. We see this as the real value add of IaaS and we have a growing army of customers that agree.
The real challenge facing the IaaS sector today is to deal with the quite valid concerns regarding data portability, user control and security that current and potential customers express. None of these issues are addressed by adopting a de factor standard be that Amazon’s or another vendor’s.
Best wishes,
Robert
—
Robert Jenkins
Co-founder
CloudSigma
Main site: http://www.cloudsigma.com
Blog: http://www.cloudsigma.com/blog
Twitter: http://www.twitter.com/CloudSigma
smoothspan said
Welcome to the discussion, Robert. Let’s drill down on that a bit:
“I think the idea that IaaS can be defined by one common standard that allows any meaningful level of interoperability is not true.”
As I said, the wonderful thing about standards is there are so many to choose from. Who said anything about having only one? However, there is more than enough critical mass of usage around Amazon, and more than enough vendors adopting their API, that there would be a “meaningful level of interoperability.” For example, if I can ship my AMI to another provider, bring it up immediately, and replicate my DB across to that other provider, then I have the potential to create a failover situation across vendors. A great many folks would consider that “meaningful interoperability.”
“The current largest player, Amazon, has key limitations which, if adopted by the whole industry, would severely limit the applicability of IaaS as a product to many sectors in particular the enterprise sector.”
As I said before, there is no reason it has to be the only standard. But it is fast becoming the de facto standard. You run a startup trying to provide some Cloud Platform Infrastructure. Let’s talk about Enterprise limitations. If your startup is the only place I can run because I’ve built on your proprietary API, what am I going to think about that? Most Enterprises are going to either ignore the proprietary Cloud or look at what it takes to make their stuff run on Amazon. Add more players like Google that support the Amazon API’s and they’re even happier. As a matter of fact, if your startup provided a superset of the Amazon API, they would feel safer coming to you too. Lastly, having a standard to point to is something the Enterprise also generally likes.
There are Enterprises today building on Amazon. It isn’t for every app, but it is Hipaa and SAS-70 and has a growing ecosystem around it of further services. Those services could immediately be available anywhere else that chooses to standardize on the API. That’s also goodness from the perspective of most Enterprises.
“Just as there isn’t one standard in traditional web hosting, there is unlikely to be a standard in cloud computing in the medium term.”
Two reactions to that. First, there is a whole set of de facto standards around web hosting that are pretty similar. They work well enough that it isn’t that big a project to uproot and move to a different web host. Second, web hosting does not impact the fabric of the software nearly as much as arbitrary server hosting. It is not simple to change to a completely different set of API’s for the Cloud, hence the desirability of standards. If we got to the point, like we have with web hosting, where de facto standards were similar enough that a port was easy, we might not need a standard. I’m skeptical given the greater complexity of the underlying problem that this will happen.
Robert, the reality is that CloudSigma would be unaffected by the standard we’re discussing if the Amazon API is so inferior and enough customers want what you’re offering. But, there is enough critical mass (measured in the hundreds of millions of dollars, vast number of apps, vast amount of storage, and size of ecosystem, not to mention vendors willing to endorse this standard) that Amazon is here today as a de facto standard for some very large markets. The market as a whole and ISV’s in particular can keep trying to fight it and lose, or it can endorse the API and potentially get a little control over it. I personally think it’s better to try to get some control and get Amazon into an open standards making process than to let it build momentum on its own. It’s largely a question of whether the Amazon API has a large enough Cloud mindshare that it’s already a standard like it or not.
Either way, it isn’t going away any time soon.
Cheers,
BW
cloudsigma said
Thanks for the response. I think your response seems to be based on the assumption that Amazon are gaining market share, they aren’t. Other newer players such as Rackspace and others with more flexible and open platforms are growing faster than Amazon. Amazon may be growing in absolute terms but they are currently losing relative to the market as a whole. Separately from this anecdotally I know many companies who have switched away from Amazon to competing providers over the last six months haven’t used them previously when there was little alternative. If you attend any event on cloud computing you will find many Amazon customers complaining about performance and reliability; personally I haven’t had such feedback regarding many competing vendors.
Secondly, as stated previously, in reality no enterprise is going to want to switch quickly and instantly to some other cloud as in the example you state. The reality is a lengthy due diligence, testing and confidence building process with any additional/alternate provider. The time that re-writing some scripts to work with an alternate API as part of this would be is less than 5% of the man hour investment in total.
As stated previously, the real cost of switching is changing system architecture not the API. Amazon actually have a relatively ‘unique’ shall we say approach to IaaS that hasn’t been adopted subsequently from other market players be that Terremark, Rackspace, GoGrid and others. The reality is that switching between the other market participants is significant easier than switching between Amazon and any other platform. Having an Amazon compatible API isn’t even close to the full story here.
Finally, the more open and flexible a platform, the less it matters where a particular user is migrating from if at all. The less requirements an IaaS platform imposes on a user, the freer they are to adopt a usage pattern compatible with their own needs. That might mean using it in a way that mimics, Amazon, it might mean using it in a way that mimics Rackspace or some other player. More likely it will mean using the cloud platform in a way that mimics how they already work internally and that is the rub. Platforms that reduce the time investment needed to migrate to them significant increase the return on investment and reduce vendor lock-in as well. As such for new market entrants in terms of customers (which is the vast majority of the potential market still), they will choose a platform based on these factors. That is why we see players other than Amazon now growing more quickly. Google adoption of Amazon’s storage API has more to do with their 0% market share than its status as a de facto standard. Google isn’t an organisation that can address the enterprise market in any of the markets it operates in and, if it chooses to enter the IaaS cloud computing space, it won’t be the provider of choice to the enterprise there to. New market entrants are choosing platforms that are more open and more compatible with their existing internal infrastructure. This also supports creation of hybrid clouds.
Best wishes,
Robert
–
Robert Jenkins
Co-founder
CloudSigma
Main site: http://www.cloudsigma.com
Blog: http://www.cloudsigma.com/blog
Twitter: http://www.twitter.com/CloudSigma
smoothspan said
Robert, you’re going to have to show me some hard facts and figures about this migration away from Amazon and loss of market share. The figures I see and the companies I talk to reflect just the opposite, so I have to give it low credibility.
“In reality no enterprise is going to want to switch quickly and instantly to some other cloud as in the example you state.”
Of course they won’t, but the reality is that having the ability to do so if needed is compelling to the Enterprise.
Robert, I don’t see you on LinkedIn. I see your company is in Switzerland, but only your CEO has linked in. You guys should sign up and show people your track records. That’s another thing that Enterprises find to be compelling.
Cheers,
BW
cloudsigma said
I’ll try and dig out the reports I read regarding this. In essence Amazon is growing very strongly but in the US (where it faces competition of the sort I was talking about) others are growing even faster aka Rackspace and others, hence the relative loss of market share. The cake is growing but Amazon’s share is shrinking. It isn’t surprising you are seeing examples of people joining Amazon, its an exploding market but more and more are choosing other vendors instead.
Regarding your other comments, this isn’t about myself or my company. Personally I don’t care for social/professional online networks as a business owner not employee but that is a whole different discussion. I’ve set out a logical exposition of why a standard API is not a relatively important factor in reducing friction between clouds. Having read your responses I’m still of that opinion. Yes being able to change clouds is ‘compelling to the Enterprise’. You are making my argument for me here. The API is not the main problem when moving cloud as outlined previously. That is why Amazon’s exotic approach is making more open and flexible platforms attractive; because they can switch more easily between clouds should the need arise. Therefore choosing the one cloud that faces the highest switching costs and which no other major market participant supports isn’t a sensible enterprise strategy.
I am happy to check back in in 6 months to see what Amazon’s market share is then. I’ll gladly eat my hat if they still have the same share they do now (it will be lower) 🙂
Kind regards,
Robert
–
Robert Jenkins
Co-founder
CloudSigma
Main site: http://www.cloudsigma.com
Blog: http://www.cloudsigma.com/blog
Twitter: http://www.twitter.com/CloudSigma
cloudsigma said
Hi Randy,
It hasn’t been six months but its interesting to see Rackspace catching Amazon in some fields. Have a look at the latest ‘State of the Cloud’ update at http://www.jackofallclouds.com/2010/07/state-of-the-cloud-july-2010/ . Rackspace are growing at 19% vis-a-vis 13% for Amazon. Also in terms of top sites (of course not the only measure), Rackspace can hardly be considered ’rounding errors’. Its quite interesting to contrast this with the same post on the market status in May 2010: http://www.jackofallclouds.com/2010/05/state-of-the-cloud-may-2010/ .
Obviously the sectors and figures are by no means ‘the full picture’ but it gives an overall impression slightly different from the Amazon domination one often portrayed.
Best wishes,
Robert
—
Robert Jenkins
Co-founder
CloudSigma
http://www.cloudsigma.com
Is the Open Stack Cloud Announcement a Big Deal? « SmoothSpan Blog said
[…] Comments cloudsigma on Amazon Web Services: The De Fa…smoothspan on Amazon Web Services: The De Fa…cloudsigma on Amazon Web Services: The De […]
Is the Open Stack Cloud Announcement a Big Deal? said
[…] why Open Stack is great. Open ain’t enough, particularly with a group arguing that Amazons APIs ought to be a standard and Amazon continuously innovating and cutting prices while many can’t seem to even get in the […]
Choosing the right cloud platform | ZDNet said
[…] all this just a matter of APIs? It depends what you need to specify and how sophisticated the APIs are. At present, the APIs […]
ssconrad said
What we found in using EC2 and S3 is that AWS does a poor job of reporting usage and costs. The only time you can really figure out what’s going on is after you’ve already received your bill. We went so far as to prototype a tool to help. Recently we made it public, so if anyone wants to take a look, give us feedback:
http://awswatch.spikesource.com