Does Agile Development Work for Sytems Integration?
Posted by Bob Warfield on September 15, 2008
The gang of Enterprise Irregulars are currently wondering whether Agile can work for implementing Enterprise Software. By implementation, I mean installation and integration of a complex package such as an SAP or Oracle ERP system.
It’s an interesting question. I thing it could work, and work well in theory, but I think there are a number of factors that would make it extremely difficult to do in practice.
First, let me say I’m a fervent believer in Agile software development, and helped contribute some of its fundamental underpinnings early in my career. One of the seminal papers behind the early Agile movement was a study done on one of my development groups that found productivity was off the charts.
With that said, Agile is not for everyone or every project. There are some requirements for it to be successful that fly in the face of a lot of the established norms for IT projects and implementations. That’s not to say you couldn’t use Agile for such projects, but unless everyone involved clearly understood the Agile requirements, it would not likely be an unsuccessful attempt.
What are these requirements? In the context of implementations, I would see the following sorts of major conflicts:
First, Agile is the opposite of a boil-the-ocean project. It recognizes that you shouldn’t even try to do everything in one release. It argues for many small evolutionary releases, or if not releases, at least milestones where things are stabilized and accepted as complete before moving on. In some cases overly ambitious goals that can only be met in a single humongous milestone are rejected.
Second, Agile recognizes that it is often folly to completely specify a system before writing any code. As such, it seeks to be iterative, to get something built, to get feedback on that checkpoint, and then to iterate until the feedback is acceptably positive. This requirement is particularly at odds with the normal desire on both sides to manage risk (apparently) by documenting everything in incredible detail before much gets done. In my mind there is a false sense of security in long SOW’s and other documents that is no substitute for signing of small subsystems that you can actually use, test, and demonstrate are working. But, this is what the world grew up on and is used to.
In many ways, Agile has a risk profile more similar to SaaS. Consider a huge up front investment in specification to be the equivalent of paying an On-premises license fee up front. Neither the spec nor the license are worth much until some software is written and working.
Lastly, Agile is not well suited to extremely large groups and it eschews heavy process quite on purpose. It’s important to note that it does this not to enhance creativity as Vinnie fears, but precisely to enhance productivity. It recognizes (as is called out in the Coplien paper I link to above) that the primary productivity bottleneck in software development is communication.
The Mythical Man Month issue is the more people, the harder it is to keep them in sync. If our bench is limited, we perforce select the best possible players to be on the team. This again, is sharply at odds with the typical SI practice of having very few talented players and lots of grunts. The talented players often move around constantly fire fighting. An Agile approach favors using only the most talented players on small teams. Clearly this interferes with the economics of most SI’s who want to bill as many hours as possible using labor that is as cheap as possible.
Dhananjay Nene said
Software development used to be boil the ocean, used to completely specify a system, and often had large team sizes before agile came on the scene. Many of these teams including some large teams did successfully migrate to agile with positive results. I really cannot see the conflicts in agile development and systems integration. I am aware of some large ERP implementations stretched out over years which might have benefited from agile development.
The issue is the nature of the contract with the customer. Agile brings the uncertainty which is often the bugbear of any complex software development (or for that matter SI) effort into the agreement with the customer (unless consciously suppressed). I suspect if the customer-vendor get into an agreement of a fixed time release with potentially variable features (as opposed to fixed feature release with potentially variable time), I really can’t see any major issues with agile per se. That is a behavioural and expectations change which many software providers and their customers went through and I suspect shall eventually happen to many SI customers and vendors as well.
The last issue you bring up regarding few vs. most talented players is again imho lesser to do with SI per se. If one has a diverse talents team, the same issue would need to be worked through with non SI development as well. I suspect we shall be hearing many more success stories rather soon, of modest talent teams benefiting from agile.
smoothspan said
Dhananjay, welcome!
The issue is precisely the contract with the customer, both real and implied. As I’ve pointed out, many customers look for boil the ocean projects that are intricately specified in waterfall style before any implementation really begins. That’s at odds with Agile, and while we can speculate about how well it might work if customers would change their ways, we also have to recall the old adage that the customer is always right. In truth there are reasons just below the surface why both customers and SI’s like this arrangement. On the customer side, we have the issue of predicatability and budgeting. They want a package, fixed price if possible, but tightly controlled if not. Even if IT understands the benefits of Agile, it is alien to most of the players that control the budget. How many CEO’s, CFO’s, VP’s of Sales, VP’s of Marketing, or VP’s or HR (to name a few that come in on large ERP or HRIS projects) understand Agile and can get behind it’s requirements? All too often they’ll see it as an excuse to get on an open-ended time and materials contract where the SI can milk them endlessly. Note: I’m saying that will be their perception, not that it is the reality.
On the SI side, it isn’t an awful lot better. The reason customers fear the open-ended time and materials deal is because so many services organizations have taken advantage of them in the past. They are, after all, largely incented to bill hours and little else.
It’s going to take some very enlightened customers and SI’s to get very far with Agile, but I suspect that if they can, more will follow.
Cheers,
BW
Dhananjay Nene said
Bob,
I couldn’t agree with you more. However while reading through your response, it suddenly struck me that one could perhaps look at the issue at a slightly different level of abstraction.
Agile development imho requires a highly trusting environment between the customer and the vendor, and what you stated above perhaps goes to the very root of the underlying issue. You can’t successfully implement agile fully without a trusting environment, and current SI relationships may not necessarily measure up to those requirements.
I wonder if customers and vendors will need to figure out a way to build better trust before being able to go the agile way where such trust doesn’t adequately exist today. Thats a bleak thought but I suspect the issue is perhaps quite real.
Dhananjay
tuomoks said
Yes, agile works in almost any project (even in very large) assuming..
Very good points about agile in replies. Agile really needs trust, maybe more than anything else. Now, agile is nothing new except as a word, maybe it has been forgotten for a while but the largest projects I did (participated) in 70’s and 80’s, as whole new insurance, banking and government(!) infrastructures, were very agile and very successful. And fast, a relative term in large projects but definitely faster and less expensive than the estimates.
Private and government are of course a little different, in private companies if a CEO says and supports the idea that everyone is a resource, it works (mostly). In government where people change and move around all the time and have more a political agenda than supporting “the common good”, an agile project has to be more limited but can be as successful.
These projects were successful because at that time (for example) I worked with many vendors and later on competing with same vendors but the trust was there. Going from IBM customer to competition didn’t change our relationship at all and we executed many projects together, for the benefit of all, vendors and customers. And even on business side we always found that teamwork / sharing with competing corporations was much better for all than trying to do everything, every time yourself. Amazing times, haven’t seen it since mid 90’s, something changed!
Now, I have a small problem with agile in some companies today. It is (again?) seen as a rigorous process model? 15 minute meetings every morning? Whatever? Same or even more strict decision, design, development, etc chain as in other models but sometimes lacking the sanity checks or even worse, the strategy – because it is called “agile”? “Old” habits die slowly!
The fact is that agile doesn’t need less subject expertise, it should allow the experts to be resources when needed and free when not needed. It is supposed to be very flexible but requires a strong discipline so resources don’t get wasted and the time can be used for productive work. Egos, power play, politics, own agendas, etc can kill an agile project very fast and this is why the CEOs and other top management has to support it fully.
It used to be an enjoyment when you knew that your CEO, customer CEO and even the competing CEO were resources when some resolutions on that level were needed. And it of course played well down the line in those companies, mostly, some middle management was sometimes a little difficult but not for long, they were told either to work or leave because the caused problems and delays!
So, agile works but (IMHO) needs some changes especially in decision making today. Nobody has time for everything and if that person is the only one with authority (and on vacation, sick, traveling, conference, other office, etc) the process is not anymore agile. There is (almost) no place for “skilled labor” in agile process, everyone has to be responsible of his/her decisions but also have the authority based on current task in hand – yes, there are risks, people overestimate their knowledge and capabilities, mistakes happen, etc but that’s also the beauty in agile – the mistakes get fixed fast because they come obvious immediately, not in next “project meeting” or even worse, in next phase, whatever, and everyone learns so risks decrease over time, a lot.
IT Project Failures mobile edition said
[…] Irregulars discussion of Agile development methods applied to software implementations, which Bob Warfield blogged. Word game image via California Energy Commission.] posted by Michael Krigsman September […]
Agile Engineering Design » Blog Archive » Agile Systems Integration said
[…] globally) but I was a bit disturbed by how little information related to this activity. Apart from one excellent post and subsequent discussion – there was little to go […]
Agile Systems Integration said
[…] globally) but I was a bit disturbed by how little information related to this activity. Apart from one excellent post and subsequent discussion – there was little to go […]