SmoothSpan Blog

For Executives, Entrepreneurs, and other Digerati who need to know about SaaS and Web 2.0.

Archive for the ‘software development’ Category

What Will GPU-On-The-CPU Mean for Analytics?

Posted by Bob Warfield on January 21, 2011

This week’s InfoBoom-sponsored post is all about Intel’s announcement that it would be shipping chips that include an integrated graphics processor (GPU) on the same chip as the CPU under their so-called Sandybridge architecture.  A lot of folks probably ignored the announcement thinking it meant better video games for their kids and perhaps their laptops, but there is a more interesting way to look at the impact of such architectures for business.

GPU’s are not just for graphics, as it turns out.  They’re extremely powerful supercomputers in their own right, with vector processing and other capabilities that are well on par with the Cray supercomputers that ruled during my college computer science days.  The Cray X-MP ran 941 Megaflops back in the day and was considered strategic weapons grade computing.  The latest Nvidia CUDA GPU’s weigh in at 1 Teraflop or better, so are fully capable of keeping up with the old Cray.  What happens when that kind of power is available on every CPU?  A whole lot of power, and a whole lot of change.

Check out my post over on InfoBoom to find out!

Posted in cloud, software development | Leave a Comment », nice name, shame about the platform (Huh?)

Posted by Bob Warfield on January 20, 2011

Matt McAdams writes an interesting guest post for Phil Wainewright’s ZDNet blog, Software as Services.  I knew when I read the snarky title, “, nice name, shame about the platform,” I had to check it out.

The key issue McAdams seems to have with (aside from the fact his own company’s vision of PaaS is much different, more in a minute on that) is latency. He contends that if you separate the database from the application layer, you’re facing many more round trips between the application and the database.  Given the long latency of such trips, your application’s performance may degrade until your app “will run at perhaps one tenth the speed (that is, page loads will take ten times longer) than a web app whose code and data are colocated.”  McAdams calls this a non-starter.

If using or similar database-in-the-cloud services results in a 10x slower app, then McAdams is right, but there are already ample existence proofs that this need not be the case.  There are also some other interesting considerations that may make it not the case for particular database-in-the-cloud services (hereafter DaaS to avoid all that typing!).

Let’s start with the latter.  Salesforce’s offering is, of course, done from their data centers.  From that standpoint, you’re going to pay whatever the datacenter-to-datacenter latency is to access it if your application logic is in some other Cloud, such as Amazon.  That’s a bit of a setback, to be sure.  But what if your DaaS provider is in your Cloud?  I bring this up because some are.  Of course some Clouds, such as Amazon offer DaaS services of various kinds.  In addition, it’s worth looking at whether your DaaS vendor hosts from their own datacenter, or from a publicly accessible Cloud like Amazon.  I know from having talked to ClearDB‘s CEO, Cashton Coleman, that their service is in the Amazon Cloud. 

This is an important issue when you’re buying your application infrastructure as a service rather than having it hosted in your own datacenter.  It also creates a network effect for Clouds, which is something I’ve had on my mind for a long time and written about before as being a tremendous advantage for Cloud market leaders like Amazon.  These network effects will be an increasing issue for PaaS vendors of all kinds, and one that bears looking at if you’re considering a PaaS.  The next time you are contemplating using some web service as infrastructure, no matter what it may be, you need to look into where it’s hosted and consider whether it makes sense to prefer a vendor who is colocated in the same Cloud as your own applications and data.  Consider for example even simple things like getting backups of your data or bulk loading.  When you have a service in the same Cloud, for example like ClearDB, it becomes that much cheaper and easier.

Okay, so latency can be managed by being in the same Cloud.  In this case, is not in the same Cloud, so what’s next?  Before leaving the latency issue, if I were calling the shots at Salesforce, I’d think about building a very high bandwidth pipe with lower latency into the Amazon Cloud.  This has been done before and is an interesting strategy for maximizing an affinity with a particular vendor.  For example, I wrote some time ago about Joyent’s high speed connection to Facebook.

Getting back to how to deal with latency, why not write apps that don’t need all those round trips?  It helps to put together some kind of round-trip minimization in any event, even to make your own datacenter more efficient.  There are architectures that are specifically predicated on this sort of thing, and I’m a big believer one whose ultimate incarnation I’ve taken to calling “Fat SaaS“.  A pure Fat SaaS application minimizes its dependency on the Cloud and moves as much processing as possible into the client.  Ideally, the Cloud winds up being just a data repository (that’s what and ClearDB are) along with some real-time messaging capabilities to facilitate cross-client communication.  The technology is available today to build SaaS applications using Fat SaaS.  There are multiple DaaS offerings to serve as the data stores and many of them are capable of serious scalability while they’re at it.  There are certainly messaging services of various kinds available.  And lastly, there is technology available for the client, such as the Adobe AIR ecosystem.  It’s amazing what can be done with such a simple collection of components:  Rich UX, very fast response, and all the advantages of SaaS.  The fast response is courtesy of not being bound to the datastore for each and every transaction since you have capabilities like SQLite.  Once you get used to the idea, it’s quite liberating, in fact. 

Surprisingly, many have seen this model, though they may not have thought much about it.  As McAdams points out, “ will do better with developers of mobile apps, which contain the user interface and the app code in the same bundle.”  Yup, many mobile apps are Fat SaaS.  The architecture becomes a lot more interesting when you start thinking about how popular apps are on mobile versus apps in browsers and why.  This also points the way towards some of the types of apps that are particularly well suited to Fat SaaS:  complex games, rich content creation applications, and a variety of things where the simple fill-in-the-form-and-do-the-workflow metaphor just isn’t right would do well with Fat SaaS.  There are also advantages for cost and scaling, where Fat SaaS it works great so long as your application’s usage patterns are largely hub and spoke.  What I mean by that is that there isn’t a lot of need to do cross-client processing in bulk.  Sure, a record may pass through several user’s clients on its journey, but you don’t have to do complex business logic that involves looking at thousands of those records across many clients very often.  When you’re doing hub-and-spoke, the client hardware is free in Fat SaaS.  It may be cheap in an elastic Cloud, but it’s hard to beat free!

The situation for aggregate processing looks worse for Fat SaaS, but it isn’t necessarily black and white.  Very often we find systems whose transaction processing behavior is a lot different than what we want for Business Intelligence.  Hence we wind up optimizing two different datastores–one the transaction processing store, the other the BI Data Mart.  The same approach can work here.

One last comment about McAdams’ post.  He concludes with a pitch for his company’s products:

The bigger criticism applies to all DaaS offerings, and it’s this: they’re solving the wrong problem. Making databases more accessible to programmers who already know how to use databases is nice and all, but how about making databases more accessible to business users? 

Why not let non-programmers build web apps without writing code? To do this, would have to include things that discloses explicitly are not part of the platform: page layout tools, reports, dashboards, and so on. These can be built with, but using requires programming knowledge.

It seems to me the more innovative cloud DB players are the companies providing a cloud database with a complete, integrated app development platform that requires no coding, only point-and-click configuration. These platforms, like TrackVia (the author’s company) or Intuit’s QuickBase, are doing more to change how cloud apps get built than the better-known DaaS vendors are.

As I’ve said in previous posts, I’m not a fan of the “anyone can program with our special point and click application” ideas.  The demos look whizzy, but this is really not innovation, and the tools are pretty limited in how far they’ll take you.  We’ve similar tools for a long time under earlier guises such as 4GL or later guises such as the PC desktop databases like dBase or Microsoft Access.  We’ve seen it in languages with UI Builders like Visual Basic or Adobe Flex.  I have to say, we’ve seen a lot more point and click programmers than we have seen DaaS.  In addition, the idea that the problem a DaaS solves is one that is already solved, in other words that they’re trying to make understanding databases easier for programmers who already understand databases is also pretty far off base (and surprisingly given McAdams’ past in DBA software).

DaaS is all about two things that many developers are usually not so good at:  Scaling and Automating Operations.  There is a lot of work required for either and it’s work that the majority of developers who may have a ton of domain expertise for their app area typically are short on.

Posted in cloud, software development | Leave a Comment »

Forget SaaS vs On-Prem, Here are 8 Application Styles to Consider

Posted by Bob Warfield on November 4, 2010

The EI discussion about Microsoft’s poor handling of Silverlight brought out the different viewpoints swinging. The “RIA was never a good idea” camp was in full force as was the “this confirms everything about HTML5” and the “Flash is on the same train as Silverlight” camps.

I don’t see these developments nor the evolution of Flash and its strategy as a confirmation that RIA is a bad idea at all, that Flash, at least is going away, or that HTML5 is the answer to all the world’s problems. Whether your RIA consists of Flash widgets embedded in HTML (my favorite RIA strategy for web apps) or AJAX HTML, RIA’s are essential for many kinds of app and HTML5 is years from being a first class choice for that work. There has been too much tendency to conflate every conceivable nuanced app type as one single web app that is best served by one single technology. That way lies crappy UX, high dev costs, and longer lead times to market.

There are new categories of application coming along all the time, and the vast majority haven’t really stopped to think about the evolution of application types and their ecosystems. Let’s forget the ongoing dogfight about Clouds, elasticity, SaaS, and Multitenant for a minute, and have a look at factoring software along some other dimensions.

I see 8 different application design centers, and you choice of which design center to use and which tools and platforms to go with it can matter quite a lot.

#1 Plain Old Desktop (POD) Apps


I always loved the term “POTS” = Plain Old Telephone Service, so I’ll borrow the idea for PODs, which are “plain old desktop” apps. This is still a huge business including MS Office and much of what Adobe sells by way of content creation tools. It’s also huge for complex games, though the platform is sufficiently separate we will want a separate category for console games which I won’t get into here.

Typical platforms include the desktop OS–Windows or Mac, and the traditional developer languages such as C# and Java. To deal with the pains of PODs, your app needs to be something that requires too much horsepower for the other architectures as well as a rich User Experience (UX).

#2 Server Software

A command line is a UX, and for some purposes, its even a good UX, so we will count this as an “app” category. It’s like the PODS except there are more languages and OS’s to consider. On the OS side we have Unix in commanding lead (in all of its many flavors) with Windows distantly to the rear. From the langauge side, the world has probably spent more time and effort building an explosion of different evolutionary language branches here than anywhere else. I can’t even begin to list them all, but suffice it to say that when in doubt, you could do a lot worse than to bet on it being a language for server side development. Certainly we can count on Java, C#, and Ruby on Rails as all being in this category.

Some languages are a little bit muddled as to whether they’re server or client languages. PHP is a little of both that mostly errs towards client in my book, but you wouldn’t think of it for server-less apps (if no client is a headless app, are those tail-less apps?). For this app category, we have no UX but a command line, so I’m not sure we’d pick PHP. Put it in the category of helping out Plain old web apps and RIA’s.

#3 Client Server


The Client Server model is tried and true and still a huge business, especially for business software. I don’t know how much genuinely new Client Server software is being built anymore. The Cloud and SaaS are a much better model for most applications. Think of Client Server as the somewhat unwieldy combination of a POD and Server Software. Most of what I’ve said for those other styles carries to this one when combined appropriately.

The combination of Desktop, Server, and Client Server are the oldest app styles that are still vigorous today. We’ll call mainframe software a flavor of Server or Client Server. All three are under siege or revolution. Desktop and Client Server are clearly under siege as various web technologies via to take their place. Server is under the dual revolutions of Cloud and Multitenant SaaS. There have been many minor infrastructure revolutions as well as the world transitions from SOAP to REST or decides POJOs (Plain Old Java Objects) make more sense than deep J2EE architectures (let alone CORBA style, Service Bus, and all the rest of that melee).

#4 Plain Old Web (POW) Apps


No fancy RIA tech, AJAX or otherwise. This is the modern equivalent of the half duplex 3270 green screen. But, it is lowest common denominator friendly. That means everyone can access it, but the experience may not be great. Save it for times when coverage is more important than satisfaction. Simple UI you just have to get through once in a blue moon is perfect.

The platforms? Who cares. POW apps should use such vanilla HTML they never notice. Tools and languages? Vanilla HTML and whatever your favorite dev tools are for that.

If we add POW apps to the other 3, you really do have the majority of revenue and installed base at present. What follows are newer models that are catching on to a greater or lesser degree. I should add that I see POW as stable and non-controversial, Server as growing but full of revolution, and the other two (Desktop and Client Server) as in decline to greater or lesser degrees.

#5 Rich Internet Apps

RIA’s are web apps that live in the browser but provide a nicer UX than a POW can offer. AJAX, Flash, or Silverlight have to be there for it to count. HTML5 supports this poorly at present, but everyone knows it wants to go here over time.

The “Rich” part of a RIA can boil down to lots of things, so let’s call some out so they don’t go unnoticed. Yes, we will started with Rich User Experience. But what does that mean?

Well, instead of posting a form, you have enough expressiveness that you can actually create new kinds of widgets and respond to user inputs with much more fluidity. We also have rich media, including sound and video to play with, as well as sophisticated ways of manipulating bitmaps to produce animations and art.

However, depending on the RIA platform, you may also experience other “riches.” Flash player delivers a very high performance virtual machine. It isn’t Java-class, but it is darned good. Recently, browser makers have decided performance matters for HTML as well, but it isn’t clear they’re keeping up with Flash. If your RIA has to do some kind of crunching, Flash may help it along. It isn’t just about the high performance VM, there is also the support for more elaborate data structures, which are also helpful where more horsepower is needed.

I have been fascinated by an idea I came up with called “Fat SaaS”.  We live in the multicore era where computers no longer get faster in raw terms, they just get more cores.  At the same time, it is very hard to write software that takes advantage of all those cores.  On the server side, the world has been evolving towards dealing with the issues need to scale large problems onto grids of commodity PC’s as a way to deal with the realities and economics of the multicore era.  On the client side, it seems that machines accrue more and more unused capacity without delivering good reasons for customers to upgrade as often as they used to.

Fat SaaS is an architecture that pushes as much down onto the client as possible and leaves the server farm largely for data storage.  In an extreme Fat SaaS case one could imagine that the clients become the business logic processing grid.  The goal is to tap into the computing resources that lie fallow there, and there are a lot of them.  Most organizations will have more client cores than server cores.

But we digress…

The last bit of “riches” I want to touch on is device independence.  We largely got there for server software, and write once run anywhere is a beautiful thing.  Other than Flash, we are nowhere in sight of it for the client.  This is a sad and embarrassing tale for our industry, and one that developers too often try to power their way through by just writing more code.  Browser incompatibilities are insidious.  As I write this article in WordPress, I’m trying to use their rich text editor.  It runs with varying degrees of success and a little differently on each browser.  Lately, it mostly fails on IE, meaning the UX has not been kept up to snuff.  If they would simply invoke Flash to do one single thing, manage the text editing, they could deliver an identical experience on every browser, not to mention many mobile devices too, though on Apple’s devices they would need an app to do that.

This browser dependence is absolutely the bane of HTML based RIA’s, and I can see no reason why the problems won’t continue with HTML5.  As vendors race to see who can deliver better HTML5 support sooner, keep eyes peeled for the incompatibilities to start.  If they do, that’s a sign that the HTML5 dream is not all its cracked up to be, even in the fullness of time.  It will take some new generation and a bunch of restarted browser implementations to get there.  Meanwhile, Flash will have gained another 5 to 10 years lease on life, despite detractors.

#6 Rich Internet Desktop Apps

RIDs!  A RID is a really cool thing.  It is the web era approach to building desktop apps–we build them with web technology.  Amazingly, nobody seems to have noticed that if you need to run disconnected, or if you want to build an app with web technology that does meaningful things on the desktop, Adobe is the last man standing.

Google killed Gears and Silverlight looks more and more deprecated by the day.  HTML5 is still struggling to get to the minimum RIA bar and will be for years.  What’s a poor RID developer to do, but focus on Flash?

RIDs are fascinating apps, and I would argue your nuts to build any new PODs or Client Server when this model would work. In the area of games, there are amazing developments in 3D coming for Flash Player that will also play to this architecture. Lastly, Ialready mentioned “Fat SaaS”, a model which is ideally suited to RIDs.  The RID just gives the Fat SaaS access to local disk and more of the machine’s facilities.  Fat SaaS work can go on whether the client is connected or not, and a connection can wait until the client needs to connect, which reduces the demand on the Cloud Server Farm still further.

#7 Mobile Apps

MAs are closely related to RID’s, and certainly much better known. The non-HTML RIA platforms are excellent for this purpose, and it is no accident MSFT sees this as Silverlight’s future.  HTML5 doesn’t do enough and it will be years before it does to have real apps stand alone with it. iPhone’s default toolset is basically foisting desktop technology on you to build these apps, and I’m skeptical this is as productive, but the stakes are high enough people deal with it.

#8 Mobile Internet Apps

MIAs:  when you want to live in a browser on a mobile device.  Plenty of data suggests that for apps that are not heavily used, users prefer to consume via browser.  This shouldn’t be controversial as it simply makes sense.  The browser makes it much easier to dip in and out of a lot of apps that you probably never use enough to learn really deeply.  MIAs are a different category than RIAs because like the RIA, there are considerations beyond one-size-fits-all HTML.  However, a MIA could be a RIA MIA (LOL) or a simple HTML MIA.  I won’t bother breaking those two out as new app types–we already have 8 and that’s enough.  The reality is the MIA is a placeholder reminder that you have to do something to make your app palatable on a mobile device lest you have a below par UX.


Have you thought about when to use each of these design centers, what the optimal tool sets are for each, and especially how to weave an all-platform strategy with as little work as possible?

Most haven’t.  I see low expertise in out in the world as much of this is new and very few organizations have so much breadth of experience.  Most of the time organizations start with one design center and move chaotically on to others as targets of opportunity present themselves.

We increasingly live in a world where being on one or a few platforms will not be good enough, and we won’t be able to build for all with organically grown “luck” strategies.  Maybe this is a good time to start thinking through these issues in a systematic way for your organization and its products?

Related Articles

Recursivity and RWWeb talking about Fat SaaS.

Posted in platforms, software development, user interface | 6 Comments »

Is Twitter Not Multitenant or What?

Posted by Bob Warfield on September 20, 2010

So here I am, 5 days after the big announcement, and still no new Twitter UI.  We just finished a weekend, which would seem to me like a logical time to roll it out to the remainder of the audience.  No joy.  WTF, over?

Is Twitter not multitenant, or what?  I sure they look with disdain at hearing that term usually reserved for business SaaS software, but I mean really, what’s up with this, guys?  Did you not get the memo about keeping all users on the same code line?  Did we forget to Tweet that somewhere along the line so you never read it?  Is your architecture so fragile that you can’t afford to let everyone have the new UI at once?  WTF is up with this?

I don’t know what’s up with me on this.  I mean, some guys who “get it” are all “Meh” about it, while others think it’s a whole friggin’ platform.  Basically, Twitter needs to be a richer medium for me to like it better, so it sounds cool to me and I want to see it. .

OK, I can see the advisability of not rolling it out to everyone in one fell swoop.  I’m trying to calm down, and sure, I’ve written about release feathering myself.  That’s kewl and all, but how long is this going to take and why isn’t there more transparency into when I as a user can expect to get the UI?  You know, like there must be an algorithm or some such.  If my handle starts with an “A” I get to go first, unless I’m a Tech Crunch reporter with the handle “Zelda” in which case I get to go first too.  Hey, at least I could figure out what to expect.  Or maybe you could even make my expected upgrade date accessible to me in the UI somewhere.

Here is the thing.  If you are going to make the biggest update ever.  If you’re going to have PR about it and all.  You’ve got to have more transparency and a shorter release feathering cycle so people know what to expect and can get access sooner.  I mean Google is much bigger and managed to get the Priority Inbox to me a lot sooner.  After all, you’re pissing off guys like Anil giving it to his wife before him and all.  Anil is right by the way in that post about it being a platform that doesn’t act like Switzerland (if you want to call that pane a platform).  But Twitter has been acting more and more walled garden-ish all the time.

Do you know what I mean?

Posted in saas, software development, Web 2.0 | 4 Comments »

Android is so Open, it got Flash Back on the iPhone

Posted by Bob Warfield on September 9, 2010

How ironic.  On the same day that MG Seigler was penning one of his characteristically snarky posts (snark is one of the ways Techcrunch pursues its Follower Economy) about how Android isn’t really open, Apple announces the return of Flash to the iWorld.

Adobe’s stock price is cooking this morning as a result (I wonder if 10% of Adobe’s revenue is even traceable to Mobile Flash? Maybe), and I have to admit, being the Adobe Flex fanboy that I am, I’m chortling over my morning cafe mocha too!

A couple of things to talk about on this news.

First, why do I award credit for this to Android?  Peeps, come on–Steve Jobs didn’t just wake up from his hissy fit about Flash (thanks for that link, Larry!) and decide, “Oh darn, I was wrong.”  No, no, no, not happening.  Something grabbed Steve Jobs by the throat and shook him right out of his reality distortion field.  That something was competition.  Not only has Android been selling extremely well, but the Bastiches have even been using Apple’s, “Hey, we’re not the Man, we’re Cool“, marketing tactics.  Dude, what’s next, Android thinking differently (I can’t see Google having the populist moxie to get that grammar wrong the way Apple did, all those PhD nerds just couldn’t handle it).

Competition is what forced this change, MG Seigler.  Android’s “We’re more open than open can be” pitch was working.  In fact, this Apple announcement sweeps away most of what one would’ve argued were barriers to openness.  It isn’t just Flash that benefits.  Besides which, a heck of a lot of people love their iDevices (3 out of 4 of our whole family have iPhones and we share an iPad) but want Flash access.  Forget movies, they can change.  I can’t even see the stock graphs on Yahoo because they’re Flash.

Gordon Gecko was wrong when he said, “Greed is good.”  It pains me as a person often described as being slightly to the right of Attila the Hun to reach that realization.  What Gordon should have said was this:

Competition, for lack of a better word, is good. Competition is right, Competition works. Competition clarifies, cuts through, and captures the essence of the evolutionary spirit. Competition, in all of its forms; Competition for life, for money, for love, knowledge has marked the upward surge of mankind.

And here, it has let Flash back into the iWorld.  Good job, Android!

Second, I want to take a few lines to talk about why I think Flash is so important.  Much has been made of how the LAMP stack has transformed web development.  Companies can now create products without requiring millions of dollars in R&D expense.  But the LAMP stack primarily benefit the back-end, in other words the server-side.  What about the client?  What is the equivalent of a LAMP stack for client development?  Unless you’re a tremendous fan of 3270 green screen UI, and I know some are even in this day and age, you need the equivalent of the LAMP stack to efficiently product great clients.  Here is the secret:  Flash is the LAMP stack of UI development!  Yes, there are some Flash wannabes out there, specifically Silverlight.  So what?  Microsoft would love for .NET to be the LAMP alternative too.  Flash is it for these simple reasons:

–  It is ubiquitious.  Aside from the iWorld, and that changes today, the vast majority of the world has already installed Flash and it runs in their browsers.  That’s 99.3% penetration in mature markets.  Essentially, everyone has it.

–  It is a write once run anywhere tool that really works.  You write your Flash code and all 99.3% of users can run it without you needing to give it another thought. 

–  AJAX and Javascript, the alternative, is riddled with browser dependencies.  Any dev team that has tried to get even simple rich UI like a Wiki-style rich text editor to work understands this.  It’s a huge overhead to keep up with all of these browser dependencies, and a constantly moving target.  With Flash, Adobe does that work for you so you can get on with building your app.

–  Flash has amazing graphical power and Job’s protestations to the contrary is actually quite fast.  People are writing 3D games in it, for example.

–  Flash can transcend the browser to create apps via AIR, and as we’ve discovered in the mobile world, apps are where it’s at.  Google used to offer Gears for this purpose, but pulled the plug on it.  Silverlight?  Hey, competition is good, bring it.  But they aren’t really there yet.

This all adds up to something that’s very important to the ongoing evolution of User Experience across all devices.  Simply put, Flash deserves to keep going and to be available on all those devices.  Thank you, Steve Jobs, for relenting, even if it took a gun to your head.  Adobe: now is not the time to rest on your Laurels.  There are some issues here and there in the House of Flash that you still need to attend to.

Posted in apple, platforms, ria, software development, strategy, user interface | 6 Comments »

Once You Have Bad Programmers, You’re Doomed!

Posted by Bob Warfield on August 12, 2010

I love that line from Paul Graham’s post about what went wrong with Yahoo:

In technology, once you have bad programmers, you’re doomed. I can’t think of an instance where a company has sunk into technical mediocrity and recovered. Good programmers want to work with other good programmers. So once the quality of programmers at your company starts to drop, you enter a death spiral from which there is no recovery.

That observation is so true, as is this one:

Theirs was not to reason why; theirs was to build what product managers spec’d.

That’s in reference to Yahoo’s totally Product Managment-driven decision process.  It has seemed to me at times like Microsoft errs dangerously on this side too.  Product Managers are essential, and can be great, but if you place in a role where their job is to hand down stone tablets for the developers to slavishly implement, it doesn’t work.  It is not empowering to the developers who have tons of great ideas, insights, and vision themselves.  But ultimately, it’s just not a good idea to have guys in Ivory Towers thinking great thoughts they have no responsibility or accountability to deliver on.  It’s certainly not much of a team effort when things work that way.  I see great Product Managers as being the ultimate Customer Advocates.  Who else can say that they spend 100% of their time understanding the company’s customers and translating that into product requirements?  The trick is to balance those requirements against other sources of requirements, for example the need to innovate and differentiate, which far more often comes with vision rather than customer feedback.  It takes both as even the brilliant Steve Jobs discovers when his visions get too much at odds with what customers want.

The final great point from the post for me was:

In the software business, you can’t afford not to have a hacker-centric culture.  So which companies need to have a hacker-centric culture? Which companies are “in the software business” in this respect? As Yahoo discovered, the area covered by this rule is bigger than most people realize. The answer is: any company that needs to have good software.

Hacker culture often seems kind of irresponsible. That’s why people proposing to destroy it use phrases like “adult supervision.” That was the phrase they used at Yahoo. But there are worse things than seeming irresponsible. Losing, for example.

Yes indeed, there are worse things than seeming irresponsible.  Though I think the appearance of irresponsibility is only an appearance and comes from suits not being able to understand the hackers.

So let’s say you’re running a company that has Bad Programmers.  Are you really doomed? 

It’s an interesting problem to think about fixing it.  How did you get there to start with?  Here are some thoughts:

–  Like Yahoo, you had a culture that didn’t value Technology.  Great programmers can smell that a mile away and will general avoid it.  I have seen cases where most of a company didn’t value the developers much, but there was an elite core group where great developers could flourish.  It’s often hard for overly sales-driven cultures to value developers as much as they should.  As one friend put it, sales-driven cultures see the Customer as God and Sales as the Church.  There were certainly elements of this at work for Yahoo.

–  You outsourced or isolated your Developers.  This is not a guarantee you have Bad Programmers, but it is a virtual guarantee they’re not really plugged into your corporate culture the way they should be.  Developers can smell this too.  Sometimes they like being off by themselves, but its too easy for it to turn into a Stone Tablet scenario at which point they’re unhappy.  The opposite extreme is the Dev Group that has no customer contact and is just implementing whatever they feel like.  Also bad!

–  The fish rotted from the head.  Bad Leadership is a problem for Developers.  They have little patience for a lot of politics or a weak leader that’s easily snowed.  Dilbert is not a recipe for a great Hacker Culture.

–  Lazy hiring.  You started out with some great developers, but you did a poor job hiring and building an organization.

–  Lousy process.  Smart people abound, but the processes in use are not productive, or worse are destructive.  This is usually a function of poor change control.  If left to fester, you can get spaghetti code from the brightest of teams.

So back to the question of fixing it, and while we’re at it, let’s also look at avoiding it in the first place.  Consider these proactive steps:

–  Make sure your corporate culture values what Developers do.  They don’t call them Software Companies for nothing.  They’re not Sales Companies, Marketing Companies, or Finance Companies.  Make sure you act like it, while also recognizing the huge value these other groups bring. 

–  Keep the Developers well plugged into your corporate culture.  If they’re remote, work overtime looking for ways to make sure they’re still plugged in.  Rotate them through corporate.  Send the right people out to the remote locations frequently.  Never let it turn into a case of micromanagement from afar (Stone Tablets) or no management at all.  Make sure there is strong leadership in the remote locations and make sure the leaders there most of all are plugged back into corporate.  Let me tell you, this will be a lot of work.  You outsourced and offshored to save money.  Welcome to the first of many hidden costs that will make it seem less attractive, though not fatally so.

–  Get great Engineering Leadership.  You need leaders who are inspired, visionary, extroverted, technical, and downright fun to hang around.  They need to be great not just with the developers, but with their peers in the other functional areas.  The leadership, more than anything, has to move the cultural values both ways across the blood brain barrier that often separates Developers from the rest of the culture.  They’d better speak both languages (Geek and Business) to do that well.

–  Realize that Hiring is critical from day one, and it never stops.  I could write many posts on hiring, but I will leave it to one simple observation.   99% of the time people seem to regard hiring as a temporary distraction from what’s really important, such as delivering a product.  But hiring the wrong people will cause you to deal with more kinds of Hell for longer than any architectural mistake you can possibly make.

–  Keep tuning the process.   Start out with the obvious Best Practices, but don’t rest on your laurels from there.  Development organizations have two responsibilities: delivering Great Products and building a superior Software Factory.  They spend most of their time focused on the Products, but like Hiring, the Software Factory is not to be overlooked.  Any given product release is just a battle in the war.  If your Software Factory releases faster than the competition, if it builds better product every cycle, and if it does this more cheaply than the competition, it’s a huge advantage.

One last piece of advice.  If you already have a lot of Bad Programmers runnning around, find a way to isolate them.  Don’t just beg Good Programmers to come in and drop them piecemeal into a pit of Bad Programmers.  They won’t be effective and they won’t hang around.  Start out making sure you have great leadership, and build new groups with Good Programmers.  Preferably do this product by product, and through acquisition of Teams that are known good entities who know how to work together to accomplish great thinks.  Don’t hire Good Programmers and term them into Bad Programmers inadvertently, or drive them away altogether.

Posted in software development | 8 Comments »

Is Your Child a Computer Prodigy?

Posted by Bob Warfield on August 2, 2010

I recently was asked (along with others), “What language a budding computer scientist should try to study in school?

Fundamentally, it’s the wrong question.  This will sound harsh, but alas, it is only reality.  Or, if you like quotes:

“I never did give anybody hell. I just told the truth and they thought it was hell.”
Harry S. Truman

The language doesn’t matter, and no self-respecting computer science curriculum should be letting you choose languages.  They may expose you to a variety, but you’re not there to learn computer languages.

Real developers are born and not made, no matter what the quality of the curriculum.  You can’t teach it, you can only help it to blossom.  If you’re one of these people, you’ll learn many languages quickly as you become interested in whatever requires a particular language.  If you’re not, you can still have a fine career in IT or Prof Svcs.  Most of the people who are really going to get it were programming before they set foot in a University.  In fact, a really good one will be tempted to skip school if you’re not careful.  Computers call to them with a Siren’s Song that cannot be ignored. 

Find a person like this a quality program, preferably one that is under the Math Science moreso than Engineering auspices.  The abstract and theoretical sides will be more nurtured there and it’s harder to pick them up by osmosis.

If you wonder whether your child has this talent, Python is a great place to start.  Get them a book called Python Programming for the Absolute Beginner and see what they can do.  If you’re one of these people, don’t push your child to it.  Watch from afar.  Be interested, and responsive, but don’t force it, and whatever you do, don’t make it a competition. 

Like all careers, you have to love it to be really good at it.

Posted in software development | Leave a Comment »

Tim Bray’s Schtick: He Likes 3270 Green Screens as UI

Posted by Bob Warfield on May 6, 2010

So I’m reading Bray’s blog as usual, and I come across his argument against Flash that I see occasionally–namely, that all Flash UI sucks.

Why?  Here are his words:

What’s not to like, then? Well, the user experience, which in my experience is fourth-rate for anything but games; No “Back” button, feaugh. And of the course the fact that it remains essentially proprietary.

So, I use a Flash-blocker every day, and I am not a friend of Flash inside Google, but none of my arguments have anything to do with being part of the Web, or not.

The closest thing to an explanation is that there is no “back” button.  Hmmm, is that really the issue?  Needing to understand more, I did a search for “Tim Bray RIA”.   Sure enough, I find this article which says:

One of his points is that Rich Internet Applications aren’t worth the hype. He says that web applications are generally better than desktop applications, because they enforce simplicity and support a back button, and that users prefer them.

WTF?  Really?  All this for a frickin’ back button?

When you get down to it, Web UI sans AJAX, Flash, or some other RIA (Rich Internet Application) engine is just like the 3270 green screens of old.  It’s fill in the form and press the “Enter” button (BTW, that’s why it’s called the “Enter” button).  Yeah sure its simple.  Darned right, it doesn’t get much simpler.  And for many things, it’s just exactly the right thing.  But heck, if its the only way to do everything we’re really selling the web short.  It’s the old when you have a hammer everything is a nail.  I just can’t get my head around the idea that “web applications are generally better than desktop applications” if for no other reason than there’s so many types of user interaction that just don’t make sense for a 3270 Green Screen.

Maybe its a Learning Style thing, I dunno, but it’s not for me.  Give me a simple UI where simple makes sense and a Rich UI where that makes sense, which is a lot more places than just games.

Posted in ria, software development, user interface | 7 Comments »

Minimizing the Cost of SaaS Operations

Posted by Bob Warfield on March 29, 2010

SaaS software is much more dependent on being run by the numbers than conventional on-premises software because the expenses are front loaded and the costs are back loaded.  SAP learned this the hard way with its Business By Design product, for example.  If you run the numbers, there is a high degree of correlation between low-cost of delivering the service and high growth rates among public SaaS companies.  It isn’t hard to understand–every dollar spent delivering the service is a dollar that can’t be spent to find new customers or improve the service.

So how do you lower your cost to deliver a SaaS service? 

At my last gig, Helpstream, we got our cost down to 5 cents per seat per year.  I’ve talked to a lot of SaaS folks and nobody I’ve yet met got even close.  In fact, they largely don’t believe me when I tell them what the figures were.  The camp that is willing to believe immediately wants to know how we did it.  That’s the subject of this “Learnings” blog post.  The formula is relatively complex, so I’ll break it down section by section, and I’ll apologize up front for the long post.

Attitude Matters:  Be Obsessed with Lowering Cost of Service

You get what you expect and inspect.  Never a truer thing said than in this case.  It was a deep-seated part of the Helpstream culture and strategy that Cost of Service had to be incredibly low.  So low that we could exist on an advertising model if we had to.  While we never did, a lot was invested in the critical up front time when it mattered to get the job done.  Does your organization have the religion about cutting service cost, or are there 5 or 6 other things that you consider more important?

Go Multi-tenant, and Probably go Coarse-grained Multi-tenant

Are you betting you can do SaaS well enough with a bunch of virtual machines, or did you build a multi-tenant architecture?  I’m skeptical about your chances if you are in the former camp unless your customers are very very big.  Even so, the peculiar requirements of very big customers (they will insist on doing things their way and you will cave) will drive your costs up.

Multi-tenancy lets you amortize a lot of costs so that they’re paid once and benefit a lot of customers.  It helps smooth loads so that as one customer has a peak load others probably don’t.  It clears the way to massive operations automation which is much harder in a virtual machine scenario.

Multi-tenancy comes in a lot of flavors.  For this discussion, let’s consider fine-grained versus coarse-grained.  Fine grain is the Salesforce model.  You put all the customers together in each table and use a field to extract them out again.  Lots of folks love that model, even to a religious degree that decrees only this model is true multi-tenancy.  I don’t agree.  Fine grained is less efficient.  Whoa!  Sacrilege!  But true, because you’re constantly doing the work of separating one tenant’s records from another.  Even if developers are protected from worrying about it by clever layering of code, it can’t help but require more machine resources to constantly sift records.

Coarse-grained means every customer gets their own database, but these many databases are all on the same instance of the database server.  This is the model we used at Helpstream.  It turns out that a relatively vanilla MySQL architecture can support thousands of tenants per server.  That’s plenty!  Moreover, it requires less machine resources and it scales better.  A thread associated with a tenant gets access to the one database right up front and can quit worrying about the other customers right then.  A server knows that the demands on a table only come from one customer and it can allocate cpus table by table.  Good stuff, relatively easy to build, and very efficient.

The one down side of coarse grain I have discovered is that its hard to analyze all the data across customers because it’s all in separate tables.  Perhaps the answer is a data warehouse constructed especially for the purpose of such analysis that’s fed from the individual tenant schemas.

Go Cloud and Get Out of the Datacenter Business

Helpstream ran in the Amazon Cloud using EC2, EBS, and S3.  We had help from OpSource because you can’t run mail servers in the Amazon Cloud–the IP’s are already largely black listed due to spammers using Amazon.  Hey, spammers want a low-cost of ops too!

Being able to spin up new servers and storage incrementally, nearly instantly (usually way less than 10 minutes for us to create a  new multi-tenant “pod”), and completely from a set of API’s radically cuts costs.  Knowing Amazon is dealing with a lot of the basics like the network infrastructure and replicating storage to multiple physical locations saves costs.  Not having to crawl around cages, unpack servers, or replace things that go bad is priceless. 

Don’t mess around.  Unless your application requires some very special hardware configuration that is unavailable from any Cloud, get out of the data center business.  This is especially true for small startups who can’t afford things like redundant data centers in multiple locations.  Unfortunately, it is a hard to impossible transition for large SaaS vendors that are already thoroughly embedded in their Ops infrastructure.  Larry Dignan wrote a great post capturing how Helpstream managed the transition to Amazon.

Build a Metadata-Driven Architecture

I failed to include this one in my first go-round because I took it for granted people build Metadata-driven architectures when they build Multi-tenancy.  But that’s only partially true, and a metadata-driven architecture is a very important thing to do.

Metadata literally means data about data.  For much of the Enterprise Software world, data is controlled by code, not data.  Want some custom fields?  Somebody has to go write some custom code to create and access the fields.  Want to change the look and feel of a page?  Go modify the HTML or AJAX directly.

Having all that custom code is anathema, because it can break, it has to be maintained, its brittle and inflexible, and it is expensive to create.  At Helpstream, we were metadata happy, and proud of it.  You could get on the web site and provision a new workspace in less than a minute–it was completely automated.  Upgrades for all customers were automated.  A tremendous amount of customization was available through configuration of our business rules platform.  Metadata gives your operations automation a potent place to tie in as well.

Open Source Only:  No License Fees!

I know of SaaS businesses that say over half their operating costs are Oracle licenses.  That stuff is expensive.  Not for us.  Helpstream had not a single license fee to pay anywhere.  Java, MySQL, Lucene, and a host of other components were there to do the job.

This mentality extends to using commodity hardware and Linux versus some fancy box and an OS that costs money too.  See for example Salesforce’s switch.

Automate Operations to Death

Whatever your Operations personnel do, let’s hope it is largely automating and not firefighting.  Target key areas of operational flexibility up front.  For us, this was system monitoring, upgrades, new workspace provisioning, and the flexibility to migrate workspaces (our name for a single tenant) to different pods (multi-tenant instances). 

Every time there is a fire to be fought, you have to ask several questions and potentially do more automation:

1.  Did the customer discover the problem and bring it to our attention?  If so, you need more monitoring.  You should always know before your customer does.

2.  Did you know immediately what the problem was, or did you have to do a lot of digging to diagnose?  If you had to do digging, you need to pump up your logging and diagnostics.  BTW, the most common Ops issue is, “Your service is too slow.”  This is painful to diagnose.  It is often an issue with the customer’s own network infrastructure for example.  Make sure to hit this one hard.  You need to know how many milliseconds were needed for each leg of the journey.  We didn’t finish this one, but were actively thinking of implementing capabilities like Google uses to tell with code at the client when a page seems slow.  Our pages all carried a comment that told how long it took at the server side.  By comparing that with a client side measure of time, we would’ve been able to tell whether it was “us” or “them” more easily.

3.  Did you have to perform a manual operation or write code to fix the problem?  If so, you need to automate whatever it was.

This all argues for the skillset needed by your Ops people, BTW.  It also argues to have Ops be a part of Engineering, because you can see how much impact there is on the product’s architecture.

Hit the Highlights of Efficient Architecture

Without going down the rathole of premature optimization, there is a lot of basic stuff that every architecture should have.  Thread pooling.  Good clean multi-threading that isn’t going to deadlock.  Idempotent operations and good use of transactions with rollback in the face of errors.  Idempotency means if the operation fails you can just do it again and everything will be okay.  Smart use of caching, but not too much caching.  How does your client respond to dropped connections?  How many round trips does the client require to do a high traffic page?

We used Java instead of one of the newer slower languages.  Sorry, didn’t mean to be pejorative, and I know this is a religious minefield, but we got value from Java’s innate performance.  PHP or Python are pretty cool, but I’m not sure they are what you want to squeeze every last drop of operations cost out of your system.  The LAMP stack is cheap up front, but SaaS is forever.

Carefully Match Architecture with SLA’s

The Enterprise Software and IT World is obsessed with things like failover.  Can I reach over and unplug this server and automatically failover to another server without the users ever noticing?  That’s the ideal.  But it may be a premature optimization for your particular application.  Donald Knuth says, “97% of the time: premature optimization is the root of all evil.” 

Ask yourself how much is enough?  We settled on 10 minutes with no data loss.  If our system crashed hard and had to be completely restarted, it was good enough if we could do that in less than 10 minutes and no loss of data during that time.  That meant no failover was required, which greatly simplified our architecture. 

To implement this, we ran a second MySQL replicated from the main instance and captured EBS backup snapshots from that second server.  This took the load of snapshotting off the main server and gave us a cheaper alternative to a true full failover.  If the main server died, it could be brought back up again in less than 10 minutes with the EBS volume mounted and away we would go.  The Amazon infrastructures makes this type of architecture easy to build and very successful.  Note that with coarse-grained multi-tenancy, one could even share the backup server across multiple multi-tenant instances.

Don’t Overlook the Tuning!

Tuning is probably the first thing you thought of with respect to cutting costs, right?  Developers love tuning.  It’s so satisfying to make a program run faster or scale better.  That’s probably because it is an abstract measure that doesn’t involve a customer growling about something that’s making them unhappy.

Tuning is important, but it is the last thing we did.  It was almost all MySQL tuning too.  Databases are somewhat the root of all evil in this kind of software, followed closely by networks and the Internet.  We owe a great debt of gratitude to the experts at Percona.  It doesn’t matter how smart you are, if the other guys already know the answer through experience, they win.  Percona has a LOT of experience here, folks.


Long-winded, I know.  Sorry about that, but you have to fit a lot of pieces together to really keep the costs down.  The good news is that a lot of these pieces (metadata-driven architecture, cloud computing, and so on) deliver benefits in all sorts of other ways besides lowering the cost to deliver the service.  Probably the thing I am most proud of about Helpstream was just how much software we delivered with very few developers.  We never had more than 5 while I was there.  Part of the reason for that is our architecture really was a “whole greater than the sum of its parts” sort of thing.  Of course a large part was also that these developers were absolute rock stars too!

Posted in cloud, data center, ec2, enterprise software, platforms, saas, software development | 5 Comments »

Ironically, Big On-Prem Enterprise Software is the Most Customer Driven Software There Is

Posted by Bob Warfield on June 25, 2009

Vinnie Merchandani thinks big Enterprise software involves building a lot of features nobody wants.  He riffs about underutilized features and wishes for a market like the iPhone’s AppStore.   He wishes SaaS vendors to use their real time visibility into what their customers are doing to give them that kind of market-driven visibility into what to build.

Vinnie and I are fellow Enterprise Irregulars, and I really enjoy his perspective most of the time, but he has it all wrong on this one.  Backwards in fact.  Sorry Vinnie!

You see, Big On-Premises Enterprise Software is more customer driven than any other kind of software there is.

“Huh, why is that and what are you talking about, Bob?”

As one person described it to me, for these sorts of companies the Customer is God and Sales is the Church.  Sales people are natural arbitrageurs.  Any time a question comes up they are rapidly computing who to say “no” to.  Is it easier to negotiate with the customer about their request, or is it easier to negotiate with the company?  The bigger the deal and the better the salesperson (and the two go hand in hand, don’t they?), the more likely it is they choose to say “yes” to the customer and “no” to the company.

All that feature bloat in those products is as a result of saying “yes” too many times to customers.  “Yes” was easier than educating them about better ways to do things, or on how while it seems like “yes” is a good answer today, tomorrow they will have enough experience with the software to realize “yes” is no longer important.

Many view the essential function of marketing and sales as finding ways to say “yes” as often as possible, no matter what the cost.  But here is the problem.  Many times “yes” is exactly what wrecked the product.  “Yes” led to the kinds of problems Vinnie wants to escape from.

Look at it this way.  Marketers are fond of saying that since everyone consumes marketing, they all think they are good marketers.  The same is true for products.  Every user wants to tell you how to fix it before they have even explained what the problem is that needs fixing.  There is a whole school of product management that holds that the way to build great products is to constantly survey end users, build great big prioritized lists of what they want, and then give it to them.  That’s how Microsoft builds its products, for example.   But just as everyone who watches movies is not Steven Spielberg, most people who think they have a great idea for how to change a product are probably not helping.

When you approach things that way, you have a lot of trees and no sign of a forest.  Microsoft is known for successful products (unfortunately more in that past than present), but not for great products.  How do companies operate that are known for great products?  Well, consider Apple, which is perhaps the diametric opposite of the democratic product design process.  They annoint a very few philosopher kings who have the vision.  They’re spurred on by the philosopher emperor-in-chief Steve Jobs until they get it right.  Insanely Great Products are the end result.  Apple’s brief fling with moving away from that approach with Soda Pop King John Sculley was a total disaster for them.  On a much smaller scale, 37Signals is a similar model.  They actually take features back out of products if they think they were a bad idea.  That’s something that is very hard to get customers to recommend very often, and they take heat for it.  But they stick to their guns and they are generally quite successful.

Operating that way leads to conceptual integrity.  You get a forest instead of just a bunch of trees.

Companies can’t afford to ignore customers, there has to be a balance.  You need people making product decisions who are enlightened enough to connect the two ends together.  They’re going to ensure a forest gets built, but they will let customers position enough of the trees to keep them happy too.  And, where a tree is being positioned that’s in the way of a vital river needed to bring water to the forest, they will object and stop that tree being planted.  Even if it means losing one customer.  Because it ensures a healthy forest that ultimately leads to more customers in the end.

It’s a tricky process.  If you have a company that either ignores customers too much, or gives in too easily, you’ll have a product that suffers for it.  Visionaries have to be good listeners, and they have to go forth among the customers to understand their needs.  They have to be good at selling their point of view too.  If you can convince a customer to buy into the vision, they’ll quit trying so hard to plant trees in the wrong places.  Steve Jobs does all that in spades.  Is it any wonder he has been so successful?

Posted in software development, strategy, user interface | 4 Comments »

%d bloggers like this: