SmoothSpan Blog

For Executives, Entrepreneurs, and other Digerati who need to know about SaaS and Web 2.0.

Archive for the ‘software development’ Category

AmazonFail Shows Data Matters Too

Posted by Bob Warfield on April 14, 2009

No software company in their right mind would change code and move it to production without extensive testing to make sure the new code wasn’t broken.  It’s a tried and true business process supported by tools like automated build (gather up all the files, compile as necessary, package, and produce a running version), source code control systems (check in and check out with auditing so all the files that go into a particular version are known and verified), and so on.  That’s all fine and well, and though I do sometimes find a company that hasn’t made it to even that basic stage of operational process, there is a level of complacency associated with having such a process working.  The AmazonFail incident shows us that there is a lot more than just program code at stake here. 

First, what was the AmazonFail incident?  It seems that Amazon suddenly started delisting Gay and Lesbian publications from their sales rankings.  This had the effect of removing the books from search according to the WSJ.  Needless to say, the incident resulted in a great hue and cry since many assumed Amazon had done this intentionally and took it as evil behavior (along the lines of Google’s “do no evil” mandate).  Before long there were charges of  online censorship, and Twitter and the blogosphere were lit up bright talking about it.  Rumors even emerged that the incident was a result of hackers.  Eventually the whole thing became known as “AmazonFail“, and there is a hashcode for that on Twitter if you want to read all about it.  As I write this, “#amazonfail” is the third most popular search term on all of Twitter.

Clearly it’s been a major PR black eye for Amazon, but what caused it?

Amazon calls it, “an embarassing and ham-fisted cataloging error.”  Ultimately it affected over 57,000 titles including books well beyond the Gay and Lesbian themes first reported.  A more detailed internal story comes to us from Andrea James of SeattlePi:

Amazon managers found that an employee who happened to work in France had filled out a field incorrectly and more than 50,000 items got flipped over to be flagged as “adult,” the source said. (Technically, the flag for adult content was flipped from ‘false’ to ‘true.’)

“It’s no big policy change, just some field that’s been around forever filled out incorrectly,” the source said.

Amazon employees worked on the problem well past midnight, and then handed it over to an international team, he said.

Doesn’t this sound just like the sort of problem that can be caused by making a minor code change and then rolling out the changed code to production without adequate testing?  First, it was a minor change.  One field was filled out incorrectly by one person in France.  Second, it created a huge problem as minor changes in code often can.  Third, the problem was not immediately obvious, nor was the cause, so it got pretty far along before it could be fixed.  Yet it wasn’t code, it was just data.

In this day and age of Cloud Computing, SaaS, and web applications, data is becoming increasingly just as critical as code.  Metadata, for example, is the stuff of which customizations to multi-tenant architectures are made of.  In that sense, it is code of a sort.  “Soft” heuristics are common in search applications that have to decide which words to ignore completely, how to weight different words, which words might be profanity (or adult content in this case), which words are positive, negative, might be Spam related, and all the rest.  That’s all critical metadata to a search engine such as Amazon’s.  There’s a lot of other critical data to consider including credit card and other privacy-related information, financial information (what if someone gets the decimal point wrong when entering sales tax for a particular area?), and so on.

Data drives modern applications so completely, that we have to think about what this means for the security and robustness of our applications.  We’re still in our infancy on that front.  Modern software will test all the data entered to be sure it doesn’t contain a malicious payload (the SQL injection attack is one way to hack by entering special data in a field exposed to users) and there are many similar low level tests that are made.  But what about the business process that ensures the integrity of that data?  How can a single individual in France create such a big problem for Amazon by changing a little bit of data?

Let’s assume for the moment that we choose to treat that data as code.  That would mean we do some or all of the following:

–  Archive all the data changes in the equivalent of a source control (also called a “version control“) system.  We’d know every version of the data that was entered, who entered it, when they entered it, and there would be comments about why they entered it.

–  The incorporation of that data into a production environment would happen only at carefully orchestrated times (called “releases”) and only after testing had been done to ensure no problems were created.

–  If a problem manifested itself, an automated system would be available to roll back the changes one or more versions until the problem went away.  This is an important step with extremely serious problems because earlier releases will have been functioning long enough that there are no known serious problems.  Rolling back gives time to find the error without exposing all the customers to it in the meanwhile.  The error is eliminated and then the updated change is tested, and rolled out again.

Does that sound hopelessly painful as a process?  It’s exactly what most software developers go through for even the slightest change to code.  I’ll admit it would be too painful for every data change.  Amazon must add, delete, or change thousands of new listings every day.  Each one can’t be a full development release cycle.  But it does seem that applications should have some safety valves built into their fabric, and into the all-important data that they rely on.  Changing a listing is different than changing some data that almost 60,000 books rely on in search.  That data should be marked as sensitive and handled differently. 

There’s lots of architecture and process work to think about in order to avoid or minimize similar problems in the future for a whole host of applications.

Related Articles

It’s the Data Stupid.  Vinnie Mirchandani offers more examples of how we get into trouble with data.

TechCrunch gets it all wrong in a guest post by Marry Hodder.  Hodder argues unconvincingly that this was all due to an algorithm gone wrong and kept secret.  What happened is precisely NOT an algorithm.

If it had been an algorithm, it would have been an unambiguous set of rules by Hodder’s own Wikipedia-quoted definition.  Algorithms are explicit for their creators, and they are not accidental.  That they may be hidden from users does not detract from their explicitness, purposefulness, or unambiguity.

To assume an algorithm is at fault, is to misunderstand what an algorithm is, or to assume to Amazon purposefully set about creating an explicit and unambiguous set of steps to discriminate against Gay and Lesbian titles.  While the conspiracy theorists may like to natter on about this theme, it ain’t so.

The problem here, and my point in this blog post, is that we assumed we could focus on the algorithm and ignore the data that fed it.  Clearly a bad assumption.  In so doing, we allowed a minor change by an individual to create a major PR problem.  If we really think it was all a conspiracy against all things Gay, why not look for more explicit evidence.  Did Patricia Cornwell’s books about Kay Scarpetta, which include some Gay themes (Scarpetta is Gay) disappear as well?  I don’t think so.

It’s been true since we’ve had computers:  garbage in leads to garbage out.  The data is just as important as the algorithms.

Posted in amazon, cloud, QA, saas, software development | 2 Comments »

Developers: Be Passionate About What You Build

Posted by Bob Warfield on December 30, 2008

I’m parsing carefully Seth Godin’s latest post on expertise and passion.  He’s asking questions like:

Should the person who runs the customer service operations at a ski school also be required to love skiing?

Can the CFO of a large church be an atheist?

Does the head of marketing at Kodak have to have a passion for chemicals?

He winds up preferring passion for what you do to passion for the product its associated with, although he freely admits he’d like both.  Isn’t it ironic he talks about Kodak as involving passion for chemicals rather than photography, BTW?

I’m left a little uncomfortable with the whole post.  It seems very un-Godin-like.  He’s usually unequivocal in his views, and yet he seems to allow for the possibility of someone who likes being a CFO but could care less about what the organization does. 

Where startups are concerned, and especially when it comes to software development, I can’t be so unequivocal.  Looking at my own personal experience running development organizations, it would be impossible for me to do that well if I wasn’t passionate about the products we were building.  Perhaps you don’t have to love skiing to make sure customer service operations at a ski school are making customers happy.  But I don’t see how you can build insanely great software if you have no rapport with the software or its users.  Software Engineering is one of those disciplines where it’s all too easy to lose the plot and become infaturated with tools, platforms, languages, patterns, and any other thing that the customers often can’t see at all.  It leads to all the wrong things, and ultimately, to poor software.

So when you’re building your development team, when you’re hiring its members, find a way to tell whether they’re passionate about what they’re building and who they’re building it for.  It’s important.  The oft-repeated mantra to hire people who are “smart and get things done” just isn’t good enough.  In fact, I would take “passionate, and gets things done” first if I couldn’t add smart into the bargain.  For a startup, you need to insist on all three.

Posted in software development, strategy | 1 Comment »

Is Computer Science Broken, and Should Computers Be Fractal?

Posted by Bob Warfield on December 14, 2008

Interesting post via Dare Obasanjo who passes on Joe Gregorio’s post “CS Broke”.

Dare seizes on one part of Gregario’s post as being the important one:

Computer Science, at a deep and fundamental level, is broken, and that applies not only to software but to hardware. One of the reasons that I have this feeling is that after programming for the past 25 years the field hasn’t really changed. The conversations aren’t any different. You could substitute ‘Windows API’ or ‘Borland CGI’ for ‘HTML and CSS’ and you’d be having the same exact conversations I had 15 or 20 years ago. We still struggle with leaks, be it memory, or file handles, or threads, or whatever. We still have race conditions. We still struggle with software that grows linearly in features but exponentially in complexity.

Obasanjo’s response makes sense:

  1. If your definition of computer science includes HTML & CSS or Win32 then you’re doing it wrong. In the words of Edsger Dijkstra, computer science is no more about computers than astronomy is about telescopes.
  2. Even if you limit the discussion to computer programming, does it also mean that civil engineering is broken because people today still have to discuss and solve the same kinds problems faced by the builders of the great wall of China or the Roman aqueducts?

But I think Dare has missed the more interesting part of Gregorio’s point, the fractal part.  It’s fair to raise the issue that computer science is no more about computers than astronomy about telescopes, but that misses a bigger point.   First, there are other sciences and disciplines that are concerned with telescopes.  If computer scientists aren’t concerned with computers, who is?  I think Dijkstra went to far with that glib dismissal.  Someone has to be concerned.

Second, telescopes are further removed from astronomy than computers are from computer science.  Yes, there are abstract and theoretical corners of computer science such as automata theory and studies of algorithms that can’t exist or make sense on today’s computers (have you got a quantum computer yet?).  But there is a also a lot of computer science that’s engineering and is concerned with what we do day to day. 

Lastly, civil engineers do discuss problems similar to the great wall of China or the Roman aqueducts, but by now those problems are much more well understood than the issues Gregorio is concerned with.  The conversations the civil engineers are having are at a much higher level with less constant reinventing of the mud bricks pulled by donkeys on sleds methods used by their Roman and Chinese predecessors. 

But is it really so bad?  And what does Gregorio mean with his answer to the problem, which is concise but cyptic:

I’ve been struggling to find a way to express this concisely. The best way I’ve found so far is to ask:

What part of the computer in front of you is fractal?

The answer is none of it, yet in nature, which has been at this game of computation for billions of years, everything is fractal. We’re doing it all wrong.

The comments on Gregorio’s post are interesting.  My gut reaction, which is that much of what Gregorio decrys is what happens when computer science is ignored and we push ahead anyway, is reflected in the comments.  HTML itself is an ugly mess as are many languages that grew rather than were designed.  I always liked Wirth’s languages (Pascal et al) much better than C and its derivatives.  One actually reflected good computer science and was in fact easier to write compilers for as a result while the other was a half step beyond a good macro assembler and sure seemed a hack.  Certainly pragmatism wins the day in many cases and C beat Pascal for various reasons though there remains a dedicated following to Delphi to this day and its creator Anders Hejlsberg is at work with Microsoft trying to make C a better place.

But this still misses the fractal point.  What would it mean for the computer in front of you to be fractal?  Wikipedia defines a fractal thus:

A fractal is generally “a rough or fragmented geometric shape that can be split into parts, each of which is (at least approximately) a reduced-size copy of the whole,”[1] a property called self-similarity.

Some of the comments on the original thread imply the web or nature itself is fractal.  There are aspects which are fractal in both cases, but in neither case is everything fractal.  Fractals are but one way of capturing complexity in an abstract way.  In the same way, I’m not sure it makes sense to insist that computers be fractal.  Fractals I suspect are an appropriate way to view and schedule massively parallel algorithms to many cores, and that may be a good literal application of fractals.  The Thinking Machine of Danny Hillis is an ideal fractal architecture because the notion of CPU and memory merge to a single entity.  I still think Hillis was just ahead of his time and we’ll get there.  When we do, it may take some sort of fractal model to make it work at all easily or well.  Hillis showed what can happen when the astronomers build the telescopes and the Thinking Machine was a fascinating architecture.

Being a software guy, I tend to focus more on software than machine architecture though.  For me, the manifestation of a fractal nature comes from layered architectures such as the network layering discussed by Tannenbaum in his great textbook Computer Networks.  There are similar layered abstractions that exist elsewhere, but we seldom see an opportunity for practitioners to reach across layers or to create their own layers.  That’s a shame because it isn’t hard to do and leads to a lot of power.  I think of Domain Specific Languages as a sort of fractal concept because we create new languages on demand for specific tasks or domains.  I have argued before that creating a language is the only way to realize the ultimate power of the computer because executing a language is really what they do uniquely.  It’s what gives them their power.  Understanding this is definitely a computer science issue.  Read Godel Escher and Bach for a wonderful discussion of the deeper meanings to be found.  You can’t read through “the eternal golden braid” and not think of things at least somewhat fractally.

Sadly, this brings me back to the conclusion that most of the time computer science is ignored and we hack on.  There are few equipped to create a Domain Specific Language that makes solving a problem easy rather than just plunging in and hacking it out.  It’s not clear to me you can teach a person meta-thinking, which is what I call the kind of thinking that boils everything down to a language.  The distant cousin of this approach has us building a framework to solve a problem, rather than a language.  That’s not nearly as good, but often we don’t even do that.

Appologies to my regular audience for having gone off on a Geek tangent.  It happens!

If, OTOH, you liked it, here are some similar writing from this blog:

The X in Software Engineering: Exactly!

Software Handyman?!?? Not at all. Software Itelf Is Math And There Are Engineers And Scientists

Is Programming Like Music or Engineering, and Must it Be Unintuitive?

Posted in software development | 6 Comments »

Does Agile Development Work for Sytems Integration?

Posted by Bob Warfield on September 15, 2008

The gang of Enterprise Irregulars are currently wondering whether Agile can work for implementing Enterprise Software.  By implementation, I mean installation and integration of a complex package such as an SAP or Oracle ERP system.

It’s an interesting question.  I thing it could work, and work well in theory, but I think there are a number of factors that would make it extremely difficult to do in practice.

First, let me say I’m a fervent believer in Agile software development, and helped contribute some of its fundamental underpinnings early in my career.  One of the seminal papers behind the early Agile movement was a study done on one of my development groups that found productivity was off the charts.

With that said, Agile is not for everyone or every project.  There are some requirements for it to be successful that fly in the face of a lot of the established norms for IT projects and implementations.  That’s not to say you couldn’t use Agile for such projects, but unless everyone involved clearly understood the Agile requirements, it would not likely be an unsuccessful attempt.

What are these requirements?  In the context of implementations, I would see the following sorts of major conflicts:

First, Agile is the opposite of a boil-the-ocean project.  It recognizes that you shouldn’t even try to do everything in one release.  It argues for many small evolutionary releases, or if not releases, at least milestones where things are stabilized and accepted as complete before moving on.  In some cases overly ambitious goals that can only be met in a single humongous milestone are rejected.

Second, Agile recognizes that it is often folly to completely specify a system before writing any code.  As such, it seeks to be iterative, to get something built, to get feedback on that checkpoint, and then to iterate until the feedback is acceptably positive.  This requirement is particularly at odds with the normal desire on both sides to manage risk (apparently) by documenting everything in incredible detail before much gets done.  In my mind there is a false sense of security in long SOW’s and other documents that is no substitute for signing of small subsystems that you can actually use, test, and demonstrate are working.  But, this is what the world grew up on and is used to.

In many ways, Agile has a risk profile more similar to SaaS.  Consider a huge up front investment in specification to be the equivalent of paying an On-premises license fee up front.  Neither the spec nor the license are worth much until some software is written and working.

Lastly, Agile is not well suited to extremely large groups and it eschews heavy process quite on purpose.  It’s important to note that it does this not to enhance creativity as Vinnie fears, but precisely to enhance productivity.  It recognizes (as is called out in the Coplien paper I link to above) that the primary productivity bottleneck in software development is communication. 

The Mythical Man Month issue is the more people, the harder it is to keep them in sync.  If our bench is limited, we perforce select the best possible players to be on the team.  This again, is sharply at odds with the typical SI practice of having very few talented players and lots of grunts.  The talented players often move around constantly fire fighting.  An Agile approach favors using only the most talented players on small teams.  Clearly this interferes with the economics of most SI’s who want to bill as many hours as possible using labor that is as cheap as possible.

Posted in software development | 7 Comments »

The X in Software Engineering: Exactly!

Posted by Bob Warfield on January 29, 2008

Thanks for this from Discipline and Punishment:

The underlying desire — that programming ought to involve rigorous, logical reasoning about clearly defined abstractions — is largely correct. The confusion arrives when the content — this ability reason precisely about very imprecise things (like the real world) — is confused with the form, some magical ‘X’ that instantly elevates any activity to programming. Unfortunately, like all idolatry, it’s not a rational conclusion so disabusing individuals of such notions is terribly difficult. But I think they all come around eventually. All disciplines that don’t participate in metaphor, that revolve around creativity in the face of severe constraints, eventually come to realize that less is more.

Posted in software development | 5 Comments »

Software Handyman?!?? Not at all. Software Itelf Is Math And There Are Engineers And Scientists

Posted by Bob Warfield on January 23, 2008

Ravi Mohan says the title “Software Engineer” is wrong because real engineers use mathematics every day.  He goes on to clarify this view slightly by saying, “if I am not using “theory” on a day to day basis, I am not *engineering* anything.”  Modeling is also, apparently a big part of what Mohan thinks Engineers do that Software Engineers do not.  He concludes by saying the title, “should something like ‘Software Maintenance Worker’ or ‘Software Handyman’, but I guess it is easier to hire someone if his job title is ‘Rodent Officer’ instead of ‘Ratcatcher'”.

It would be easy to chalk the article up as a linkbaiting exercise more so than a thoughtful discourse, and it is, but the topic deserves a deeper examination.  The issue of whether Software is science or art comes up a lot, and I’ve written about it myself, so I have some views.

Mohan is a bit vague about whether this applies to all software engineers, or the subset engaged in writing business or enterprise software.  He relies heavily on Raganwald’s discourse here, which is largely focused on business software.  It’s important to review the Raganwald article, No Disrespect, because it offers a decidely different view than Mohan portrays.  Mohan takes the view that Reg argued in the article that programmers are clerks.  He’s missed the subtlety.  Some programmers are clerks just as some engineers in other professions are clerks.  The question Reg poses is whether you want to be a clerk?  If not, then you need to rebel a bit about how your professional situation is treating you and how it operates.  Reg points out that if you’re going to act like a clerk, you’ll be treated as one and shouldn’t complain.  In fact, the article Reg suggests if you want to understand the value of a CS Degree is quite interesting.  In it, Brian Hurt talks about an awful lot of things that sound exactly like what Ravi Mohan says real engineers do.

Mohan’s article made me laugh thinking back on my own Computer Science education.  It was under the auspices of the Math Sciences group at Rice University, and if you think we didn’t do math, you’re mistaken.  When I think about the kinds of math we did, that brings me to my next thought on the subject.  The argument can be made that traditional math is of use to Software Engineers.  Reg starts down that path himself in Raganwald’s response to Mohan.  It’s accurate, but I’ll leave that response to others. 

Here is my, somewhat more radical response:

Software itself is math.

I defy the Ravi Mohan’s to give me a formal definition that shoots that down.  Software is math.  Code is math.  Whether you are from the school that says math is symbol pushing, or the school that says math is giving a notation to intuition, there is a home for you in software.  How could you read something like Godel Escher and Bach and not come away knowing that software is math?  Certainly those that get down and dirty with a detailed understanding of algorithms, programming language semantics, or even lambda calculus (gee, its called a calculus for a reason, no?) must see that this is math.

So I think Mohan is, in the end, tragically wrong in his view that Software Engineers (yes, I am entitled to call them Engineers once again) don’t use math.  In fact, whether you understand software as math is perhaps the defining question of whether you are a Computer Scientist or a Hacker (no pejoratives on the latter).  But this is true in the other Engineering professions too.  Many Engineers are cookbook engineers.  They don’t derive the math.  They don’t create new math.  They just apply the math.  These other disciplines have their cookbook approach.  There is a list of the different kinds of bridges: truss bridges, suspension bridges, yada, yada.  Likewise we have lists of algorithms and design patterns.  Not many engineers ever create a fundamentally new bridge and not many Software Engineers create a new algorithm or design pattern.  So where’s the rub?

Getting back somewhat to the original motivation, which was to slam Enterprise software as being a land of clerks: okay, they have a point.  It can be that way.  The clerks do still use math, but it is mostly arithmetic with a little algebra and no calculus or even trig, metaphorically speaking.  However, it doesn’t have to be that way, and there can be significant competitive advantage when an organization moves away from that point of view.  Ironically, it goes back to the math and “oftware itself is math” issues.  One of the things that separates the Enterprise Clerks from Enterprise Software Engineers is an ability to reach both hands deep into the malleable stuff that is software and mold more exciting architectures.  SOA scratches at the surface, but is a step beyond clerkdom.  I don’t want to get off on a tangent defining how to do exciting Enterprise architecture, but I can tell you it can be done and it is worth doing.

The last point I’ll make is to point out:  isn’t it interesting that many of the newest languages and trends are even “mathier”?  Closures, anyone?

Posted in software development | 11 Comments »

Is Programming Like Music or Engineering, and Must it Be Unintuitive?

Posted by Bob Warfield on January 8, 2008

Two different blog posts hit me today with one of those unconscious knee jerk desires to disagree.  First was Joel Spolsky wishing that undergraduate computer science programs would quit spending so much time on formalisms and become more like a Julliard music program.  Second was Raganwald contending that if a language is really going to push the envelope of “better”, it must by necessity be even less intuitive.  Taken individually, I probably could have swallowed these pills without comment, but wouldn’t you know it: both wound up back to back in my feed reader.  Somehow they tickled the same mysterious place in my subsconscious, but I had to actually write this diatribe to understand how.

Let’s start with Raganwald.  On the face of it, the logic is pretty sound.  It’s not unlike one of my favorite quotations, which goes:

The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself.  Therefore all progress depends on the unreasonable man. 
George Bernard Shaw

The essential thesis that Reg works from is that all programming languages are equally powerful, because they’re Turing complete.  As such, he wants to talk about what makes one language better than another.  He concludes at one point that since all languages are equally as powerful, and since better is subjective, that better languages make you a better programmer, which in and of itself is subjective.  Also, making you a better programmer by definition means moving you out of your comfort zone.  Because of all that, “better” languages have to be unintuitive.  “QED”, as we used to say.

Right at this point I’ve had too many almost-rights go by to be able to sit still.  First, “better” does not have to be subjective.  We could choose to quantify it in some way.  I know that as a profession, programmers have a tremendous love hate relationship with various metrics to the point where we largely throw up our hands and dismiss them all as useless.  Sorry, but I am not in that camp.  I think there are many useless metrics, but not all are useless any more than all evaluations of every programming language are hopelessly subjective personal opinions.  I am even happy to accept fairly simple-minded quantifications such as being able to write similar things in fewer lines of code.  And please, don’t waste our time with silly contrived examples where fewer lines of code result in backwards progress.  In general, and with appropriate subjective oversight to weed out the silly off-point examples, I am convinced that fewer lines of code is a good thing.  Here is a harder one:  how about fewer clearer lines of code?  That’s going to be harder to quantify as a metric, and I can accept that. 

Now here is where I guess Reg and I part company:  is a language that needs fewer clearer lines of code necessarily less intuitive?  Reg seems to say “yes”, that language will be less intuitive.  I say that by the very construction, i.e. the lines are clearer, the answer must be “no.”  Is the only resolution that it is impossible to create a language using fewer clearer lines of code?  Gosh, I hope not, because that seems to say we can’t build better languages unless you want to quarrel with “fewer” or “clearer” not being better.  That’s my “QED”, but let’s tease things apart further, because it gets interesting.

If nothing else, this quantification business reflects a lack of good formalisms being available for a lot of squishy things like clarity of code.  There is another phenomenon at work too with this word formalism.  BTW, you can substitute “model” there if it helps make what I’m saying clearer.  There may be no single formalism or even a small set that is best for all problems solved by all programming languages.  Rather, we may have different fairly independent domains with happy little language ecosystems that work best in one or a few, but that may be terrible in others.  I am surprised and amused at the apparently fractal nature of this formalism by domain concept.  Whenever we think we’re close to finding the universal formalism with which to create the best possible language, we build something like Lisp.  After it’s built, and we use it for a while, what we discover is that such universal formalisms are really good at is creating domain-specific languages, but that they are not so good at many other things.  So in a sense, they’re lousy tools for any particular problem,  but they make it easy to create a special tool that gives the best possible solution to the sub-domain.  See what I mean by fractal?  There’s something very cool about that fractal idea that just feels right.  If you agree, it probably means we’re communicating.

Is it any surprise that it takes many formalisms to make the disparate problems we face soluble with fewer clearer lines of code?  I’m not surprised.  I’ve fiddled with just a few domains and languages in my career, and there are many differences.  Here are some I’ve played with: 

–  Database problems like the inherently parallel and set theoretic languages including the rather homely but oh-so-practical SQL. 

–  Bootstrapping everything from nothing can be done very easily with Forth.

–  Certain kinds of graphics cry out for the purity of mathematical languages with an ability to manipulate matrices and deal with collections of graphical entities.  Some like Fortran for math, but I’ve always like APL and spreadsheets.  I never played with Mathematica enough, but it is likely another alternative.  Tying something like APL and spreadsheets together with graphics in a suitably expressive way would probably make for a wonderful graphics language.  I’ve certainly seen interesting graphics come out of Mathematica, so maybe it’s been done.

–  User interface is a surprisingly parallel problem space dealing with events.  Most frameworks make this painful.  I’ll bet there is an opportunity for a great language in this niche.  I like Adobe’s Flex a lot, but it is far from perfect.  Constraint oriented programming (Thinglab!) is closer, so I am very hopeful about Adobe Thermo raising the bar further.

–  Low-level systems programming of memory allocators, process schedulers, and the like loves languages from the C family.   The less successful branch that included things like Modula-2 and Eifel is not bad either.  There are likely more recent developments I’m not familiar with.

–  Application domain specific programing likes Ruby. 

–  String processing is another world of fascinating special-purpose formalisms such as regular expressions and weird parser generator languages.

–  Creating new languages and pure computer science concepts seems to benefit hugely from Lisp. 

All of the above are sharply affected by the choice of accompanying frameworks and libraries, so they have to be considered as a whole.

This is why I’m so convinced that polyglot systems are the way to go.  Whether the polyglot family has some underlying “assembly language” like Lisp or even Ruby is almost immaterial.  The virtual machine can also be the underlying unifying theme, so long as the various polyglot languages can communicate with one another. 

A last comment on this way of thinking is that I feel Design Patterns are simply ways of imposing or adding those formalisms a particular language doesn’t directly support or doesn’t support well.  There’s been a lot of back and forth on this in the blogosphere, and it seems controversial.  Some say that a perfect language would need no design patterns.  I don’t think so.  I hope I’ve pointed out adequately that the domains require different languages so that there are no “perfect” languages.  From my perspective, you could take any design pattern and come up with a language construct that builds the pattern into the language.  Whether it makes sense or not is another issue.

Languages can become too boroque and obtuse when they are burdened with too many formalisms.  This is particularly true if the formalisms are at war with one another.  Too many ways to do the same thing may not be helpful.  I’d better stop, because I’m rapidly heading back to my polyglot programming soapbox.  Give me a set of very concise and relatively small langauges for well-defined problem domains.  Perhaps I’ll take one “general purpose”  (Jack of all Trades, master of none?) language to fill in the gaps.

Which brings me to Joel Spolsky’s post.  He wants to forget a lot of the formal training and immerse undergrads in large programming projects coordinated by talented teachers.  Joel is responding to a lamentation that schools are too quick to press kids into Java and don’t teach enough of the old formalisms (now you see why I’ve used that peculiar word so much!).  His answer is to create what he views as a “Julliard” style curriculum where the students build relatively complex software in teams under the watchful eye of talented teachers and without so much of the formalisms.  In this model, students would sign up to build a real piece of software of some kind, perhaps a game or social network.

I had one of those formal computer science educations.  I only have the Bachelor’s degree, but I completed all of the work for a PhD as well.  I just felt it would be more fun to write a business plan for my first startup than a thesis (something which maybe Joel and I agree on given his post).   We covered the formalisms in spades.  There was also an opportunity to take on some large software projects, but that was perhaps 1/3 of the curriculum. 

You are probably familiar with the observation many have made that the top 5% of programmers are perhaps 20 times more productive than the average programmer.  I know this to be true, and have seen it many times.  What does this have to do with the discussion?  Simply this: the amount of experience does not seem to matter in determine whether you’re in that 5%.  It does not seem to be possible to teach one who is not a 5% performer how to get there.  I won’t try to prove that here, take my word for it, or at least ask yourself what it means if it is true.

What is more relevant is my observations on watching that 5% group.  What did they do better?  I used to say the thing that really set them apart was a facility with factoring of all kinds.  Those were the words I used, but factoring has so many meanings these days that I feel I should clarify.  The top performers were really good at flipping problems around and viewing them from many angles.  The top players could do it from extreme “meta” positions that were layers of abstraction away from the core problem.  These people lived and breathed isomorphisms, whether or not they had any idea what the word meant.  I’ve described the application of isomorphisms to creativity in an earlier post on “The Medici” effect.  It’s worth a reread.

By contrast, the average performers tended to view everything according to a very limited set of models or formalisms they grasped.  They would try to hammer every problem into one of those few models.  In the worst cases, the models they were able to understand and employ were limited simply to basic structured programming constructs.  With difficulty they could get control flow.  They were not especially good at breaking things into functions or procedures, let alone modules.  Object oriented programming was often a bridge too far. 

We had several classes where this distinction became obvious.  One was a survey of programming languages that included Lisp, APL, SNOBOL, Prolog, and a couple of others I’ve forgotten.  We learned each new language and had to complete a model program that demonstrated we had really grasped the new models and formalisms being offered.  Another was a course wherein we built up an entire language using the fundamental Turing constructs.  The language created was our own, and everybody’s was different, but we had to build it up from the basics of Turing.  This course was done in Lisp.  It was a 300-level undergraduate course, but it was a definite weed-out course that showed who had it and who didn’t.  The last one was the hardcore algorithms course.  Once again, to understand a full range of interesting algorithms required applying many different formalisms.

You’ve probably guessed my problem with Joel’s approach.  It’s probably fine if you are seeking to create a vocational school for factory programmers.  But, the formalisms are the real meat.  It isn’t the lines of code.  The programmers who got the formalisms could grind out more code than any half dozen of the others combined.  They did so almost automatically and unconsciously.  I suspect Julliard doesn’t work quite as Joel envisions either.  Do students there stretch their thinking about musics “formalisms”, or do they spend most of their time perfecting a single arrangement and composition?

People, go out and learn some new languages.  Learn some new design patterns.  Dig out those formalisms.  It is surprising how intuitive many of them will be.  The more you pick up, the more intuitive future ones will be.  Divorce yourself from a particular language.  That’s what’s holding you up and making things unintuitive.  The best news is that because you don’t have to write reams of code ala Joel, you can learn faster and in your spare time.  The emphasis is on variety more so than volume.


Posted in platforms, software development | 4 Comments »

Coté’s Excellent Description of the Microsoft Web Rift

Posted by Bob Warfield on January 2, 2008

Coté’s latest RedMonk post perfectly captures my reservations about Microsoft, which I refer to as their “rift with the web”.  Here are the relevant passages:

Microsoft frameworks are plagued by lock-in fears. That is, you’re either a 100% Microsoft coder or a 0% Microsoft coder. Sure, that’s an exaggeration, but the more nuanced consequences are that something intriguing like Astoria will play best with Microsoft coders, unlike Amazon’s web services which will play well with any coder.

This thing he calles “lock-in fear” and the extreme polarization (encouraged by Microsoft’s rhetoric, tactics, and track record) that you’re either all-Microsoft or no-Microsoft is my “web rift”.  I’ve written about this a couple of times before, and it never fails to raise the ire of some Microsoft fan or other.

This particular RedMonk post is chiefly concerned, it seems to me, with making sure that Microsoft’s Astoria doesn’t disappear under the continuous din of innovation Amazon is putting up for us lately around their cloud computing services.  The essential new thing about Astoria in Coté’s mind is that it is a RESTful framework rather than a .NET framework:

If you’re just coding to a URL, that’s not quit so bad as coding to a .Net library and all the Microsoft baggage and tool-chain needed to support that.

I agree, but I think Coté has obfuscated two issues together that don’t have to be: the issue of how software components communicate versus whether a solution is hosted/SaaS or not.  One can imagine components communicating RESTfully without any need to have them be hosted in SaaS fashion.  My own bias would be to go ahead and host, but it’s not a requirement and to put the two together as one is conflating the issue (don’t you love that word “conflate” that has drifted into common use here in the valley?).  Coté’s contention that hosting itself will reduce fears of lock-in is also pretty hard to swallow.  While I am again a whole-hearted advocate, giving your software over to a hosted environment and all of its attendant API’s (RESTful or not) is a big step towards lock-in, no matter how you look at it.  This is again an area where Microsoft’s old school monopolist behavior won’t serve it well.  There will be fear, perhaps unreasonable, that Microsoft will take unfair advantage if handed the keys to your kingdom by hosting on their cloud infrastructure.  The problem is that they’ve failed to conduct themselves as the Swiss do in matters of banking.  They are voracious competitors and seemingly always will be.  It isn’t enough for them to win, others must lose.

There are several other interesting points made in the post.  For example, on the issue of hosting, Coté wonders why big companies are so slow to launch.  It’s very true, we’ve seen it for all the big players.  The answer is perhaps that they have more to lose and are more likely to reach the win-lose decision point too quickly and with too much momentum to gracefully correct the problems.  Startups have a distinct advantage in this.  It’s not clear to me why more big companies don’t fund startups expressly to deal with the issue with the intention of acquiring them later if things work out.

Coté also makes a hugely important point about the value of self-service:

My sense is that unless it’s all delivered as a URL with dead-simple docs and pricing (check out the page for SimpleDB), any given technology won’t work out at web-scale.

Put another way, these new technologies need to be completely self-service. If a developer has to ever talk with a human from the company or team offering the project, something has gone wrong.

Self-service is a crucial part of viral growth potential in today’s world.  Any company that releases a product without at least a semblance of a plan for how to make it self-service at some point down the road is laying the foundations for failure.

Related Articles

Latest Microsoft Office service packs decommits support for Older File Formats: Especially Competitors

This is typical of Microsoft’s “we don’t have to be nice, we’re the phone company” (appologies to Lily Tomlyn) behavior.  To access your old files requires you to delve into the registry.  Microsoft claims these old files are a security risk.  It’s tacky, it’s more lock-in, and it’s more evidence that MSFT is up to their same old tricks.

Posted in amazon, platforms, saas, soa, software development, strategy | 8 Comments »

Eventual Consistency Is Not That Scary

Posted by Bob Warfield on December 22, 2007

Amazon’s new SimpleDB offering, like many other post-modern databases such as CouchDB, offers massive scaling potential if users will accept eventual consistency.  It feels like a weighty decision.  Cast in the worst possible light, eventual consistency means the database will sometimes return the wrong answer in the interests of allowing it to keep scaling.  Gasp!  What good is a database that returns the wrong answer?  Why bother? 

Often waiting for the write answer (sorry, that inadvertant slip makes for a good pun so I’ll leave it in place) returns a different kind of wrong answer.  Specifically, it may not return an answer at all.  The system may simply appear to hang. 

How does all this come about?  Largely, it’s a function of how fast changes in the database can be propogated to the point they’re available to everyone reading from the database.  For small numbers of users (i.e. we’re not scaling at all), this is easy.  There is one copy of the data sitting in a table structure, we lock up the readers so they can’t access it whenever we change that data, and everyone always gets the right answer.  Of course, solving simple problems is always easy.  It’s solving the hard problems that lands us the big bucks.  So how do we scale that out?  When we reach a point where we are delivering that information from that one single place as fast as it can be delivered, we have no choice but to make more places to deliver from.  There are many different mechanisms for replicating the data and making it all look like one big happy (but sometimes inconsistent) database, let’s look at them.

Once again, this problem may be simpler when cast in a certain way.  The most common and easiest approach is to keep one single structure as the source of truth for writing, and then replicate out changes to many other databases for reading.  All the common database software supports this.  If your single database could handle 100 users consistently, you can imagine if those 100 users were each another database you were replication to, suddenly you could handle 100 * 100 users, or 10,000 users.  Now we’re scaling.  There are schemes to replicate the replicated and so on and so forth.  Note that in this scenario, all writing must still be done on the one single database.  This is okay, because for many problems, perhaps even the majority, readers far outnumber writers.  In fact, this works so well, that we may not even use databases for the replication.  Instead, we might consider a vast in-memory cache.  Software such as memcached does this for us quite nicely, with another order of magnitude performance boost since reading things in memory is dramatically faster than trying to read from disk.

Okay, that’s pretty cool, but is it consistent?  This will depend on how fast you can replicate the data.  If you can get every database and cache in the system up to date between consecutive read requests, you are sure to be consistent.  In fact, it just has to get done between read requests for any piece of data that changed, which is a much lower bar to hurdle.  If consistency is critical, the system may be designed to inhibit reading until changes have propogated.  It take some very clever algorithms to do this well without throwing a spanner into the works and bringing the system to its knees performance-wise. 

Still, we can get pretty far.  Suppose your database can service 100 users with reads and writes and keep it all consistent with appropriate performance.  Let’s say we replace those 100 users with 100 copies of your database to get up to 10,000 users.  It’s now going to take twice as long.  During the first half, we’re copying changes from the Mother Server to all of the children.  The second half we’re serving the answers to the readers requesting them.  Let’s say we can keep the overall time the same just by halving how many are served.  So the Mother Server talks to 50 children.  Now we can scale to 50 * 50 = 2500 users.  Not nearly as good, but still much better than not scaling at all.  We can go 3 layers deep and have Mother serve 33 children each serve 33 grand children to get to 33 * 33 * 33 = 35,937 users.  Not bad, but Google’s founders can still sleep soundly at night.  The reality is we probably can handle a lot more than 100 on our Mother Server.  Perhaps she’s good for 1000.  Now the 3-layered scheme will get us all the way to 333*333*333 = 36 million.  That starts to wake up the sound sleepers, or perhaps makes them restless.  Yet, that also means we’re using over 100,000 servers too: 1 Mothers talks to 333 children who each have 333 grandchildren.  It’s a pretty wasteful scheme.

Well, let’s bring in Eventual Consistency to reduce the waste.  Assume you are a startup CEO.  You are having a great day, because you are reading the wonderful review of your service in Techcrunch.  It seems like the IPO will be just around the corner after all that gushing does it’s inevitable work and millions suddenly find their way to your site.  Just at the peak of your bliss, the CTO walks in and says she has good news and bad news.  The bad news is the site is crashing and angry emails are pouring in.  The other bad news is that to fix it “right”, so that the data stays consistent, she needs your immediate approval to purchase 999 servers so she can set up a replicated scheme that runs 1 Mother Server (which you already own) and 999 children.  No way, you say.  What’s the good news?  With a sly smile, she tells you that if you’re willing to tolerate a little eventual consistency, your site could get by on a lot fewer servers than 999.

Suppose you are willing to have it take twice as long as normal for data to be up to date.  The readers will read just as fast, it’s just that if they’re reading something that changed, it won’t be correct until the second consecutive read or page refresh.  So, our old model that had the system able to handle 1,000 users, and replicated to 999 servers to handle 1 million users used to have to go to 3 tiers (333 * 333 * 333) to get to the next level at 36 million and still serve everything consistently and just as fast.  If we relax the “just as fast”, we can let our Mother Server handle 2,000 at half the speed to get to 2000 * 1000 = 2 million users on 3 tiers with 2000 servers instead of 100,000 servers to get to 36 million. If we run 4x slower on writes, we can get 4000*1000 = 4 million users with 4000 servers.  Eventually things will bog down and thrash, but you can see how tolerating Eventual Consistency can radically reduce your machine requirements in this simple architecture.  BTW, we all run into Eventual Consistency all the time on the web, whether or not we know it.  I use Google Reader to read blogs and WordPress to write this blog.  Any time a page refresh shows you a different result when you didn’t change anything, you may be looking at Eventual Consistency.  Even if you suspect others changed something, Google Reader still comes along frequently and says an error occured and asks me to refresh.  It’s telling me they relied on Eventual Consistency and I have an inconsistent result.

As I mention, these approaches can still be wasteful of servers because of all the data copies that are flowing around.  This leads us to wonder, “What’s the next alternative?”  Instead of just using servers to copy data to other servers, which is a prime source of the waste, we could try to employ what’s called a sharded or Federated architecture.  In this approach, there is only one copy of each piece of data, but we’re dividing up that data so that each server is only responsible for a small subset of it.  Let’s say we have a database keeping up with our inventory for a big shopping site.  It’s really important to have it be consistent so that when people buy, they know the item was in stock.  Hey, it’s a contrived example and we know we can cheat on it, but go with it.  Let’s further suppose we have 100,000 SKU’s, or different kinds of items in our inventory.  We can divide this across 100 servers by letting each server be responsible for 1,000 items.  Then we write some code that acts as the go-between with the servers.  It simply checks the query to see what you are looking for, and sends your query to the correct sub-server.  Voila, you have a sharded architecture that scales very efficiently.  Our replicated model would blow out 99 copies from the 1 server, and it could be about 50 times faster (or handle 50x the users as I use a gross 1/2 time estimate for replication time) on reads, but it was no faster at all on writes.  That wouldn’t work for our inventory problem because writes are so common during the Christmas shopping season. 

Now what are the pitfalls of sharding.  First, there is some assembly required.  Actually, there is a lot of assembly required.  It’s complicated to build such architectures.  Second, it may be very hard to load balance the shards.  Just dividing up the product inventory across 100 servers is not necessarily helpful.  You would want to use a knowledge of access patterns to divide the products so the load on each server is about the same.  If all the popular products wound up on one server, you’d have a scaling disaster.  These balances can change over time and have to be updated, which brings more complexity.  Some say you never stop fiddling with the tuning of a sharded architecture, but at least we don’t have Eventual Consistency.  Hmmm, or do we?  If you can ever get into a situation where there is more than one copy of the data and the one you are accessing is not up to date, Eventual Consistency could rear up as a design choice made by the DB owners.  In that case, they just give you the wrong answer and move on. 

How can this happen in the sharded world?  It’s all about that load balancing.  Suppose our load balancer needs to move some data to a different shard.  Suppose the startup just bought 10 more servers and wants to create 10 additional shards.  While that data is in motion, there are still users on the site.  What do we tell them?  Sometimes companies can shut down the service to keep everything consistent while changes are made.  Certainly that is  one answer, but it may annoy your users greatly.  Another answer is to tolerate Eventual Consistency while things are in motion with a promise of a return to full consistency when the shards are done rebalancing.  Here is a case where the Eventual Consistency didn’t last all that long, so maybe that’s better than the case where it happens a lot. 

Note that consistency is often in the eye of the beholder.  If we’re talking Internet users, ask yourself how much harm there would be if a page refresh delivered a different result.  In may applications, the user may even expect or welcome a different result.  An email program that suddenly shows mail after a refresh is not at all unexpected.  That the user didn’t know the mail was already on the server at the time of the first refresh doesn’t really hurt them.  There are cases where absolute consistency is very important.  Go back to the sharded database example.  It is normal to expect every single product in the inventory to have a unique id that lets us find that part.  Those ids have to be unique and consistent across all of the shards.  It is crucially important that any id changes are up to date before anything else is done or the system can get really corrupted.  So, we may create a mechanism to generate consistent ids across shards.  This adds still more architectural complexity.

There are nightmare scenarios where it becomes impossible to shard efficiently.  I will over simplify to make it easy and not necessarily correct, but I hope you will get the idea.  Suppose you’re dealing with operations that affect many different objects.  The objects are divided into shards naturally when examined individually, but the operations between the objects span many shards.  Perhaps the relationships between shards are incompatible to the extent that there is no way to shard them across machines such that every single operation doesn’t hit many shards instead of a single shard.  Hitting many shards will invalidate the sharding approach.  In times like this, we will again be tempted to opt for Eventual Consistency.  We’ll get to hitting all the shards in our sweet time, and any accesses before that update is finished will just live with inconsistent results.  Such scenarios can arise where there is no obvious good sharding algorithm, or where the relationships between the objects (perhaps its some sort of real time collaborative application where people are bouncing around touching objects unpredictably) are changing much too quickly to rebalance the shards.  One really common case of an operation hitting many shards is queries.  You can’t anticipate all queries such that any of them can be processed within a single shard unless you sharply limit the expressiveness of the query tools and languages.

I hope you come away from this discussion with some new insights:

–  Inconsistency derives from having multiple copies of the data that are not all in sync.

–  We need multiple copies to scale.  This is easiest for reads.  Scaling writes is much harder.

–  We can keep copies consistent at the expense of slowing everything down to wait for consistency.  The savings in relaxing this can be quite large.

–  We can somewhat balance that expense with increasingly complex architecture.  Sharding is more efficient than replication, but gets very complex and can still break down, for example. 

–  It’s still cheaper to allow for Eventual Consistency, and in many applications, the user experience is just as good.

Big web sites realized all this long ago.  That’s why sites like Amazon have systems like SimpleDB and Dynamo that are built from the ground up with Eventual Consistency in mind.  You need to look very carefully at your application to know what’s good or bad, and also understand what the performance envelope is for the Eventual Consistency.  Here are some thoughts from the blogosphere:

Dare Obasanjo

The documentation for the PutAttributes method has the following note

Because Amazon SimpleDB makes multiple copies of your data and uses an eventual consistency update model, an immediate GetAttributes or Query request (read) immediately after a DeleteAttributes or PutAttributes request (write) might not return the updated data.

This may or may not be a problem depending on your application. It may be OK for a style application if it took a few minutes before your tag updates were applied to a bookmark but the same can’t be said for an application like Twitter. What would be useful for developers would be if Amazon gave some more information around the delayed propagation such as average latency during peak and off-peak hours.

Here I think Dare’s example of Twitter suffering from Eventual Consistency is interesting.  In Twitter, we follow mico-blog postings.  What would be the impact of Eventual Consistency?  Of course it depends on the exact nature of the consistency, but lets look at our replicated reader approach.  Recall that in the Eventual Consistency version, we simply tolerate that we allow reads to come in so fast that some of the replicated read servers are not up to date.  However, they are up to date with respect to a certain point in time, just not necessarily the present.  In other words, I could read at 10:00 am and get results on one server that are up to date through 10:00 am and on another results only up to date through 9:59 am.  For Twitter, depending on which server my session is connected to, my feeds may update a little behind the times.  Is that the end of the world?  For Twitter users, if they are engaged in a real time conversation, it means the person with the delayed feed may write something that looks out of sequence to the person with the up to date feed whenever the two are in a back and forth chat.  OTOH, if Twitter degraded to that mode rather than taking longer and longer to accept input or do updates, wouldn’t that be better? 

Erik Onnen

Onnen wrote a post called “Socializing Eventual Consistency” that has two important points.  First, many developers are not used to talking about Eventual Consistency.  The knee jerk reaction is that it’s bad, not the right thing, or an unnecessary compromise for anyone but a huge player like Amazon.  It’s almost like a macho thing.  Onnen lacked the right examples and vocabulary to engage his peers when it was time to decide about it.  Hopefully all the chatter about Amazon’s SimpleDB and other massively scalable sites will get more familiarity flowing around these concepts.  I hope this article also makes it easier.

His other point is that when push comes to shove, most business users will prefer availability over consistency.  I think that is a key point.  It’s also a big takeaway from the next blog:

Werner Vogels

Amazon’s CTO posted to try to make Eventual Consistency and it’s trade offs more clear for all.  He lays a lot of good theoretical groundwork that boils down to explaining that there are tradeoffs and you can’t have it all.  This is similar to the message I’ve tried to portray above.  Eventually, you have to keep multiple copies of the data to scale.  Once that happens, it becomes harder and harder to maintain consistency and still scale.  Vogels provides a full taxonomy of concepts (i.e. Monotonic Write Consistency et al) with which to think about all this and evaluate the trade offs.  He also does a good job pointing out how often even conventional RDMS’s wind up dealing with inconsistency.  Some of the best (and least obvious to many) examples include the idea that your mechanism for backups is often not fully consistent.  The right answer for many systems is to require that writes always work, but that reads are only eventually consistent.


I’ve covered a lot of consistency related tradeoffs involved in database systems for large web architectures.  Rest assured, that unless you are pretty unsuccessful, you will have to deal with this stuff.  Get ahead of the curve and understand for your application what the consistency requirements will be.  Do not start out being unnecessarily consistent.  That’s a premature optimization that can bite you in many ways.  Relaxing consistency as much as possible while still delivering a good user experience can lead to radically better scaling as well as making your life simpler.  Eventual Consistency is nothing to be afraid of.  Rather, it’s a key concept and tactic to be aware of.

Personally, I would seriously look into solutions like Amazon’s Simple DB while I was at it. 

Posted in amazon, data center, enterprise software, grid, platforms, soa, software development | 6 Comments »

Not Everyone Can Program and Not Every Programmer is Great

Posted by Bob Warfield on December 2, 2007

Reg Braithwaite wrote a column recently called, “The Optimistic View”.  He’s tired of organizations pushing lowest common denominator tools because they’re afraid lowest common denominator programmers will hurt themselves (well, hurt their code really) with stronger tools.  He suggests:

But maybe, just maybe, most programmers rise to the occasion when challenged. Maybe instead of catering to the bottom 10th percentile we can be open-minded about what the 50th to 75th percentile are capable of achieving, much less the top quarter.

It’s an interesting thought, but I confess I haven’t seen an awful lot of evidence it’s true.  I have certainly been surprised by programmers before.  But the surprise has been in just how fast a really talented “junior” programmer can take their place in the ranks of the really good architects.  I have not seen a lot of evidence of the much less talented stepping up to deal with some of the real rocket science of our trade.

He gets a little more detailed in his thinking here:

Spreadsheets taught us that anyone can program. Maybe we should open our minds to the possibility that although not everyone does program in this manner today, they are capable of it if we stop telling them that they aren’t.

And when we say “Anybody can program,” we say that without reservation or disclaimer. We don’t mean anybody can program provided they are banned from using certain langauges, or anybody can program provided we hide the details of relational databases behind ORMs, or anybody can program provided they are not required to learn any idioms beyond those authorized by the local High Priest. We mean anybody can program, period, without being lashed to a heavy yoke and driven through the fields under the whip.

This is the point where we part company.  Spreadsheets did teach us that anyone can program, BUT, they did so by doing some of things he suggests we quit doing with less talented programmers.  They hide lots of details that one would have to know to accomplish the same thing using normal programming.  It is quite instructive to think about what spreadsheets do for us.  Largely, they allow us to quit thinking abstractly and organizationally, two big skills that programmers must possess.  In other words, they eliminate the need to think like a programmer. 

Consider the business of organization, or specifically of ordering.  Spreadsheets let their users examine the trees and the forest just emerges magically, and without much planning.  Tell the spreadsheet the relationship between a few cells using a simple formula.  You never have to worry about what order you do that in.  The spreadsheet does a topological sort and figures out the “natural order of recalculation.”  Because of this, there are no loops to think about.  Control structures are pretty straightforward and are seldom nested.  A simple “IF” is about all there is, and easily 90% of spreadsheets don’t use them. 

There’s a lot more clever hiding that happens as well.  Consider the whole user interface issue.  Spreadsheets give you that for free using their grid metaphor.  It’s amazing what can be accomplished there, but it isn’t really programming.  Perhaps most importantly, it doesn’t try to be.  Most of that look is fixed at design time and is not dynamically programmable. 

Data is another biggie.  For those who are stuck on static typing, spreadsheets have none, nor would it make them any “safer” to offer it.  But there are also no data structures.  Each cell stands more or less alone, with the exception of ranges (or blocks as they came to be called) referenced in formulas.  There is almost no ability to go further save Excel’s confusing array formulas which I’ll wager are used in much less than 1% of cases, and most of those were forced as a way use the LINEST builtin.

The interesting questions are how much of this can be done for conventional programming and how would one go about it?  The first issue is seriously missing.  It’s not clear we understand and agree just where the problems are.  I’m certainly not going to claim to lay it out here, but I will say this.  Layering more and more complexity on is not the answer.  That’s not how spreadsheets got where they are in terms of enabling more programming.  Adding a bunch of syntax around typing may have value for some things (chiefly making the tools better in my mind), but it isn’t how you help people.  If you want to help lowest common denominator people, figure out how to eliminate or radically reduce complexity.  Eliminate abstraction.  Eliminate the need for the peculiar kinds of thinking great programmers must do.

Spreadsheets, BTW, are a graphical dataflow paradigm.  They really are visual programming, just not using boxes and arcs.  They are also constraint oriented programming.  I could go on, but spreadsheets don’t prove everyone is a programmer.  They do something quite different that is even more useful: they make it possible to write programs even if you can’t think like a programmer.

Posted in software development | 9 Comments »

%d bloggers like this: