SmoothSpan Blog

I’m a day late, it was 20 years ago yesterday that Dr Dobbs published James Coplien’s article on how my Quattro Pro team was building software at Borland.  Jim sent me a very nice note of reminder on it:

This is a good occasion for me to tell that story of how (with Jim’s article!) I helped start the whole Agile/Scrum thing going.

The article came about because Coplien was studying software development productivity while he was with Bell Labs.  He was interviewing various groups, measuring their relative productivity, and trying to figure out what the most productive teams were doing differently.  At the time, Borland was very much in the throes of launching a slew of products that were brand new code bases for the then very new Windows platform.  I agreed that it’d be fine for him to come in and study our team’s efforts.

I didn’t know an awful lot about how other companies were developing software, heck I didn’t know too much about how Borland was doing it either except that our team seemed to be smaller than a lot of the other ones out there (including many Borland teams) and that we seemed to be more nimble (avoiding the use of that loaded word “agile”).

I was very young back then, having started my first company straight out of grad school at Rice University.  I never did get that PhD in Computer Science because instead of writing a thesis, I decided to write a business plan instead.  One thing I will say for Rice is that it was a great education in CS.  It was also the genesis of my ideas about how to build software.  One of the most interesting things that went on there was they brought in guest lecturers who gave us some impression of what was happening elsewhere in the Computer Science world at large.

I vividly remember the first “famous” software developer I met as a guest lecturer.  It was Stu Feldman from Bell Labs who had invented a program called “make”.  We all used it constantly to discover which files had been changed and speed up the compilation process so that only those files had to be recompiled and they were recompiled in the right order.  This is so basic few even talk about it anymore, but back then it was worth noticing.

What was interesting about Stu’s talk was that what he had to say about “make” was a side I had never even considered.  It was obvious to me how it worked when I saw it first, but Stu was there to talk about programmer productivity moreso than “make,” and what he had to say really caught my attention.

Programmer Productivity was a pet of mine at the time, and I had visions of going forth in the world to create the world’s greatest programming tools–tools that would radically increase every developer’s productivity.  My first startup actually built some tools like that, but that’s a story for another post.  Suffice to say that when Feldman went down that path, he had my undivided attention, because he said “make” had come about because he was tilting at the same windmills I wanted so badly to attack.

Feldman boiled the central problem in productivity down to very simple terms.  Essentially what he said was that we didn’t know how to get very many people successfully working together on a project.  After maybe 10 or 12 developers, adding more generally degraded the group’s overall productivity more than the new developer’s additional productivity could help. The central problem, according to Feldman, was communication.  “We just don’t know how to get more than 10 or 12 communicating together effectively nor how to keep them all on top of what’s going on so they could do their jobs well.”

Of course I immediately went forth after that seminar and started to soak up any wisdom I could about these issues.  Brooks Mythical Man Month (you can’t use 9 women to make a baby in a month) was also a huge influence.  In the course of reading such things, it became clear to me that software tools were not enough.  What we needed were processes and cultural changes to manage those communication problems.  This realization led me to some very basic conclusions that have informed the culture and methodologies I have used for software development ever since.  They were alive and well when Coplien analyzed the Quattro Pro team.  In fact, they were 4th generation by the time James got to see us in action.

I haven’t changed these things much.  I go through an Agile or Scrum book every so often.  They make sense, but I don’t know that they’ve influenced me to change much.

Here are the fundamental tenets I follow from these experiences:

1.  Keep teams to 10’ish or smaller in size.  I’ve never really tried to push this envelope much.  In fact, not too long after Coplien’s work came out, I wound up in charge of all Engineering at Borland and one of the things I did was to insist on all teams fitting within this parameter.  Specifically, we cut back to 10 developers, 10 QA, and 3 writers on every project.  That included consultants and part-timers.  It was met with a high degree of skepticism and emotion as you would imagine, but it worked.  We cut costs and kept all the projects within the schedules that had been set before we scaled back.  The exception to the rule is projects where you can create solid API firewalls between components and put 10 developers on either side of the firewall.  Classic for Enterprise Software projects.

2.  If you can only have 10 on the team, they’d better be the very best that you can find.  Sure seems obvious to me, but I am continually surprised at how seldom it is followed for all the wrong reasons.  Get the best you can find means you’d better be prepared to do a lot of things:

–  Make engineering a first class part of the company.  You’ll often hear me saying, “They’re called Software companies, not Sales, Marketing, or Finance companies.”  If developers are second class citizens, they’ll smell it a mile away and won’t come near.

–  Pay what it takes.  Yeah, I know, every HR group on the planet will wave a bunch of studies saying compensation is not a motivator.  They’ll tell you, “We like to pay at the 70th percentile,” if you’re lucky and it’ll be 50th if you’re not.  Most studies of most things describe very eloquently a regression to the mean.  If you’re happy being on the mean, do what they have to say.  If not, ask yourself what you’re going to do differently.  I had the whole nth percentile pay conversation with probably my favorite PR professional in the industry.  This individual is probably also the most famous for various reasons.  Eventually I had to go to the CEO of the company to make the point about why we had to pay what we had to pay and after that, I believe this individual added it to her bag of tricks.  The bottom line is that the developer may not care about any specific bonus.  But they know they have the talent and they know when they’re being paid less when others of similar talent.  That’s a very bad thing to let happen, trust me.

–  Give them a high performance environment to work in.  Today, cubicles and often desks packed as tightly as possible together are being pushed as the ultimately productivity tools.  That’s horse hockey.  Writing software requires laser-like focus and concentration.  I am all about communication, that’s how I came up with all this, but communication needs to be engineered into the process and not slapped on by osmosis.  Maximize the distractions for these kind of people and you will minimize their productivity.  I can’t tell you how many places I’ve worked that just don’t get this.  Typically, you have to cheat it into the system with a liberal work-at-home policy.

–  Parallel Career Tracks for Pure Techies and Management.  This is another one that non-developers don’t get.  The worst days on the job for someone like me are when one of their very best individual contributors asks for a one on one, comes in all tense, and announces they’re ready to be made a manager.  I always start that discussion out by asking them why they’re attracted to the job.  If I get any variation on the flavor of, “I want to make the decisions because I have the most talent and will make the best decisions,” that individual gets counseled out of being a manager.  If we reach a point where they insist or they’re leaving, well, I tell them I will miss them.  Better to lose a great architect that turn them into a lousy manager who will no doubt piss of their team and then you lose a lot more.  As I said, those are among the worst days in my career when those things come up.

3.  Great teams build great software relatively cheaply.  Nothing is more expensive to a company than having a lousy team build lousy software.  Where am I going with this?  Forget outsourcing, offshoring, consultants, and all the rest of the silly things companies do to try to save a dime here or there on software development.  Lots of very smart people argue with me on this one constantly, but it’s true.  Look, if you’re hiring all the uber talented 10 person teams you can lay hands on and going offshore is just a way to hire even more (Google, you know who you are), it’s fine.  But if you can build anything with 10 or fewer great engineers, how much savings potential is there really in going offshore versus how much risk?  Which really awesome products in this world were built entirely offshore or outsourced.  And if communication is the biggest challenge affecting developer productivity, why would you put one team half a world away, make sure there were tons of cultural disconnects, make sure there were time zone challenges, and then just to be double darned sure you interfered with communication as much as possible make them talk over the two tin cans and a string called Skype to save a few more nickels?

4.  With only 10 on the team, they can know everything that’s going on.  OK, now we’re onto communication principles.  With only 10, they can all sit in a room together and know everything that’s going on.  Why not have them do that, every single day?  They’ll help critique each other’s work.  They’ll get fired up and empowered.  They’ll know things that the others don’t and share them.  They’ll all be on the same page, for goodness sake.  Wait.  Oh my, I hear the complaints and jeers.

Meetings are such a time sync.

Yes they are if you can’t manage one effectively.  What, did you think that you Mr Manager, were in that meeting solely so that  your subordinates could make you more efficient?  Hell no.  You’re in that meeting to make your team more efficient.  Use an agenda, keep the meeting short, make sure it starts and ends on time, change up the format to make sure people pay attention, assign everyone homework for the meeting so it doesn’t go stream of consciousness, make everyone present something, and when all else fails, make it fun.  Build the camaraderie.

I’m an extroverted management by walking around kind of guy.  Having meetings like this every morning was my attempt at letting everyone share the joys of that.  It was a crucial part of our productivity, and today it is a cornerstone of Agile/Scrum where it is called a “Standing” meeting or a “Stand Up” meeting.  Apparently some folks took it a bit literally and started doing the meeting without chairs.  That’s fine, I think I’ll keep the chairs in my meetings, but they’re short enough that not having chairs shouldn’t be a problem.  I used to tell people to get them to the meeting on time that there’d be two fewer chairs than attendees, but that was just my lame joke and I don’t recall actually every doing that.

5.  If communication is the challenge, focus the communication, make it count, and minimize unnecessary communication that just sucks up bandwidth.  The best way to do that is laser-like focus that fiercely attacks each challenge and finishes it before moving on to the next.  Thus was born the Agile Backlog on my teams.

I’m a compulsive list maker.  I always thought of them as ToDos, I made them long life documents, and I constantly evaluated priorities, deferring as much as possible until later.  Imagine my surprise the first time someone asked to see my “Agile Backlog” and I had no idea what they were talking about.  After a laborious explanation, I trotted out the “ToDo” for that project and they were invariably overjoyed.  Nothing like fancy terminology to elevate an obvious idea to sainthood, lol.

My role in the meetings was to present the near-term portion of my Todo, um, I mean the Agile Backlog, to the team.  Keep them focused at 2 week intervals was and is my mantra.  Keep dragging the highest risk most poorly understood items to the front of the Backlog so they get hammered down and solved.  That latter I got from a book on military strategies and was credited to Rommel, the Dessert Fox, who always believed on focusing his guns on his most dangerous adversary on the battlefield, moving to the next most dangerous, and so on.

6.  Make it clear with pictures.  UI Mock ups and Storyboards beat pages of narrative time and time again.  People hate PowerPoint, but that’s usually because they didn’t consider the alternative of reading a 70 page document that accomplishes the same purpose.  I am a big believer in building the UI first to this day, so we can argue usability and understand what the underlying engines will be called on to do right from the start of a project.

7.  Code Reviews.  I always made my developers present their major subsystems twice.  First was an architectural review before much code had been written.  Here’s the problem we’re trying to solve, here’s how we’ll solve it, here there be dragons, here we’re not so worried.  This was done in the morning meetings and everyone chimed in with thoughts. Everyone got to hear the underlying assumptions and could squawk if they saw a bad one.  The second meeting was a demo meeting when whatever it was got to “Beta” quality and a “Here’s what worked and what didn’t” if there was no demo.  The what worked and what didn’t often wound up being a dive into the code.

8.  Interactive Middle-Out Programming.  That curious phrase is one I had way back in my distant past.  I was greatly enamored and possibly warped by the first computer language I ever learned:  LISP.  That phrase referred to building up the “bones” or what I called the scaffolding as much as possible before adding the meat.  Don’t insist that everything be finished before you connect it all up.  Connect it up with dummy responses at first and then build out the most critical and highly traffic portions next.  Fill in the details after that.  Most of all, keep something working every single build.  This was the precursor to the Continous Release mindset that we see today.  It’s the beginnings of Minimum Viable Products.

I always presented roadmaps to my management and to customers.  My goal in doing so was twofold.  First, I wanted them to get used to the idea that releases are small potatoes.  Don’t get hung up on what’s in a release, worry about the longer term vision.  Most customers are extremely understanding of that and much more willing to take a lesser product now if they’re sure you’ll eventually get where they need you to be.  Second, I always asked a lot of prioritization questions in those meetings and took every opportunity to defer things that didn’t seem high priority to later releases.  We did this right from my very first startup, so I guess we were early to the MVP bandwagon.

9.  Embrace Change.  Software is the most malleable medium in the engineering world, yet it can be brittle when timeframes are too short.  What you can’t do is ignore the inevitability of change.  We learn new things every day that can result in change.  We can fear change, or we can embrace it as opportunity.  So many aspects of the methodology I’m describing help you to embrace change when you need to:

–  Keeping the Backlog short-term focused means you can churn the heck out of the part the team hasn’t visited yet with impunity.  Re-prioritize.  Add.  Delete.  Most of all, Learn and React.

–  Middle Out Programming surfaces weakness in the interconnecting structure, the architecture, sooner.  So does UI prototyping.  Ignore the details that are not costly to change and focus on validating what is.

–  Small Teams of Uber Developers are inherently more able to change.

10.  Communicate Face to Face.  Make any documents as temporary as face to face communications.  In fact, make them only as needed to facilitate the face to face.  The software is the only long-term document to worry about.

11.  Lead, Don’t Manage.  Sell, Don’t Tell.  Everyone has to be sold.  Your team, the business side, and the customers.  Don’t hide out and expect the final product to be so dazzling you won’t have to sell.  Don’t hide behind your title secure that your people have to do what you tell them.  As soon as you get the idea, go to all your sounding boards.  As it resonates, grow that audience rapidly.  Make each meeting a learning experience on how to sell the next audience even better.  When people are sold, they become passionate.  Nothing is more valuable to the team than passionate talent.

12.  Don’t Prematurely Optimize, But Manage Technical Debt Consciously.  Middle out programming encourages the avoidance of premature optimization.  Always code it up as quickly as you know how.  But know as best you can where you think changes will have to be made.  And in so knowing, know how hard those changes will be.  If the write bones are there, the changes are confined inside particular internal organs.  If we have to change all the bones, every organ will be impacted. You don’t want Technical Debt in your bones because it becomes pervasive in a hurry.  A little technical debt in the organs is fine and can be dealt with quickly when the need is proven.

13.  Do Post Mortems and then Experiment.  After every release, we’d do a post mortem on our productivity.  What worked, what didn’t, what were the problems.  Then we’d ask for ideas on how to change.  We’d pick one or two of those and implement them on the next project.  Then, at that project’s post mortem, we would evaluate their efficacy and decide whether to continue the practice or not.

14.  Be a Software Factory, Not a Release Workshop.  I like this one to be the capstone of the principles I’ve used to guide my teams over the years.  You want everyone on the team as well as the rest of the company to think of the team not in terms of each release, but in terms of their efficacy as a Software Factory.  No competitive war is won with a single release.  No market is dominated by a single release.  Rather, it is the ability of the team to be a better Software Factory than any they compete with that wins the day.  If they can turn the release crank faster, if they can anticipate what customers want better, if they can deliver higher quality, and if they can do some combination of that more cheaply than the competition, they’re doing everything that can be expected of a Software Development Team to win the war.

Many types of investment in Software Development have to be understood in these terms because they can never be justified for a single release.  If your CEO insists on being exclusively release focused, he is going to have you mostly chasing the puck instead of skating to where the puck will be.

Today’s Agile:  Lean Manufacturing

I’ve followed those 14 tenets almost from my start.  They were certainly completely baked into my software development efforts by the Quattro Pro days, and I haven’t found much need to add to them since.

I can’t decide whether it’s ironic or inevitable, but today I am doing many of the same things I had focused on for developers for the CNC Manufacturing world.  Coplien and Scrum Founder Jeff Sutherland comment that Scrum was inspired by the Toyota Production System.

I knew nothing of Toyota when I created my methodologies (and have carried on without spending too much time on Agile or Scrum either).  I did what made sense to me based on the influences I’d had as I’ve explained.  But my bootstrapped company, CNCCookbook, is intimately involved with the CNC (Computer-Controlled Machine Tools, 3D Printers, and the like) Manufacturing World, so it was probably inevitable that I would eventually write a series of articles on Lean Manufacturing.

The series is not finished yet, but all the key concepts are there.  I can absolutely see similarities between Agile/Scrum and Lean Manufacturing, but with that said, I find them more cousins than siblings.  The Toyota work was done quite some time ago.  It’s lessons are still quite relevant to manufacturing, but I believe that what we (I) have learned about Agile methodologies are relevant to manufacturing.  CNCCookbook is currently developing a new product called G-Wizard ShopFloor that will bring that to light in a modern collaborative environment that emphasizes these hybrid methods.

The first “grainy photos” from ShopFloor are starting to come into focus, the UI prototype is done, we’re starting to show it to a few friendlies and the interactive middle out programming process is fully underway.  I’m having a lot of fun with it and can’t wait to see if the manufacturing world can benefit as much as software development has from these techniques.

I’m really having fun with it!

The other thing you might be interested in is my course on how to Work Smarter and Get Things Done.  It’s over on my Entrepreneurship site, BobWarfield.com, and it teaches how to use Agile/Scrum along with some custom software I wrote to radically boost your individual productivity.