Where Does the Slippery Slope Start?

So where do projects go wrong? If you are like me, you have been on a lot of projects and they always seem to be going along pretty well, then all of a sudden it starts to crumble. Deadlines begin to loom, hours get longer, and weird things start happening in the code that never happened before. As if that weren't enough excitement, the client finally pops into reality and starts to actually look at what it is he is paying for and hates it...claims it is NOTHING like what he asked for and starts demanding that dozens of things be changed without affecting the schedule or the budget. Before you know it, you are working until midnight trying to get the database piece to match the business piece and both of them to match the UI.

Can anyone say where the slippery slope started? There are many possible culprits. For example, the requirements for most projects are woefully incomplete. Typically the person(s) designing the software are relying on the person(s) who create the requirements to know what they want and to express it in a way that fully conveys their desires. This is almost always a bad assumption. So one possible initial slide down the slippery slope of a software project crash could be incomplete requirements. Another one could be incomplete designs, inaccurate estimates of effort to implement it and reliance on new or unproven technologies or techniques.

Actually the question of where most software goes wrong will be answered very differently depending on who you ask. A business analyst may blame incomplete requirements while a programmer may blame clueless managers. Testers may blame programmers for not fully understanding the requirements or not unit testing their code while managers may blame unrealistic timeframes and deadlines by the customer. Clients may blame the whole development organization, who may blame the whole mess on a lack of reliable funding. The fact of the matter is that there are plenty of fingers pointing and plenty of people being pointed at when a project starts to unravel, and nobody seems to be able to agree on where it started going wrong or what the root cause is.

At this point I would like to put forth the hypothesis that there is always a root cause of a project failure and it always starts at a predictable, avoidable juncture in the project. In fact, I propose that a project is doomed to failure (partial or full) before the project even starts.

As I have hinted in earlier articles, I think that Configuration Management is a critical part of any software project. Unfortunately I don't think the traditional view of Configuration Management can make or break a project. Not that projects with good configuration management aren't more likely to pull through without crashing, but I feel that there are many things that typical views of CM miss. In fact, it seems a bad idea to me to even fall back onto the phrase "Configuration Management" to describe the concepts I intend to cover. Instead, a new term is needed which covers the full spectrum of controlling all of the variables of an ongoing project. That term is one that I coined: Integrity Control.

Unlike the traditional view of Configuration Management, which tends to focus on procedure and checklists, Integrity Control is a holistic view that has a single goal: Ensure that a project remains in a known (preferably known good) state.

This is not to say that procedures are not important, but in my experience Configuration Management tends to pile on layers of procedures that, in the end, become the focus and the purpose and the real goal is lost in the shuffle. Eventually there is a subtle shift from "Are we managing the configuration of this project" to "is the change control board meeting regularly and are they publishing their meeting minutes?" Like many religions, the purpose is lost behind the layers of tradition in the form of forms, minutes, and matrices. Occasionally these procedures result in a well managed project, but very often they just add to the workload of everyone involved without any tangible benefit.

I know that I will hear from some readers who will say "there is no tangible benefit because the procedures are not really followed...they are pencil-whipped and done just for the record. If the same people would actually follow the procedures, the results would always be predictable and reproducible!”. Maybe. But that misses the point. The point is that, from what I have seen, that doesn't happen very often in the real world. Whether or not it would, in theory, work as designed matters little when the implementation of such a process taxes the people doing the work far beyond any perceived benefit. In the end they fill out the forms and follow the workflows because they are required to. They create a trail that, if audited, can prove that they process was followed. It is a subtle difference that can't be pegged to one incident of person, sort of a collective wink and nod that everyone participates in. The “this is just useless bureaucracy” becomes a self-fulfilling prophecy as time is spent on a process that is being followed only for the record instead of time being spent on actually getting the work done, which reinforces the participants' perception that it is all just a big waste of time.

So just make people do their job. They are getting paid to follow the process, so make them. Problem solved, right? If that were the case, there would be far fewer projects in crisis mode and many more being completed on schedule and on budget. Numerous industry surveys have shown that this isn't the case though. Even among projects that follow rigid methodologies (CMMI, Six-Sigma, ITIL, Agile, whatever) tend to drift into the rocks. I have seen just the opposite. The harder you push developers to do “process stuff”, the more they resist. In one case they proved their point by following the process to the letter and the entire project came to a grinding halt.

I believe that the reason this happens so often is not because developers are a difficult bunch or because they a too focused on writing code to do the rest of their job. I believe that it happens because the people doing the work are really smart. The average software developer can spot a pointless procedure in a millisecond and can find a thousand subtle ways to avoid following it that can't be spotted by even the sharpest-eyed manager. In most cases, there isn’t even any hard evidence that the process is being avoided. But it is.

In short, when smart people are forced to do a lot of work with little or no perceived gain, they are certain to find ways to game the system. To illustrate my point, I want to relate a story that I know is true because I was there at Fort Gordon, GA in December, 1992 to participate. I call this one:

Rudolf the Red Faced Drill Sergeant

The US Army does not have the reputation of being the brains of the military, but there are many, many very smart people serving our country in the US Army. I know because I had the pleasure of serving with some of them in the Satellite Communications field. The training for the Strategic Satellite Communications Technician job was a very intense year-long technical school that started with basic electronics and progressed through extremely complex communications theory, protocols, transports, and routing. It was a tough school and, not to be tooting my own horn (there is a point to this), only the very sharp survived the full year. The only military school with a higher attrition rate was at the time the infamous Navy Nuclear School.

There was a problem with the schools though. The students were managed by combat-trained drill sergeants. I am not implying that all “drill instructors” (as they insisted on being called) are mentally-challenged, but ours pretty much were. They knew we were smart and they spent many long nights dreaming up tactics to keep us “in line”.

Army Basic Training is tough on some folks. Personally I didn’t find it very stressful. I can endure pretty much anything for two months and I was very good at gaming the system. The problem came in when we got to the year long SATCOM school. The Army was supposed to ease up on us a little in technical schools and let us concentrate on learning. Instead they put drill sergeants in charge who made no secret of their disdain for us “bunch of smart privates”. The idea that we should be kept in line was something that most of them shared and lived for. They would impose new rules constantly, do surprise inspections, call formations at all hours of the night, and generally use basic training tactics to keep us from getting away with anything. Or so they believed.

But we were smart. With each new rule that was dreamed up, we quickly found a handful of loopholes and workarounds that got us back to the level of freedom that we wanted. After several months of this type of treatment, beating the system becomes a survival tactic and, in our case, a game. I have heard of prisoners of war who find ways to get anything they want and do pretty much whatever they want to do right under the guards’ noses, and this is they type of mentality that developed at the SATCOM school. It came down to “Adapt or collapse”, which several of my fellow soldiers ended up doing. Several simply left the Army. Walked out and went home in the middle of the night. Several went crazy, some tried to kill themselves by various oddball means, and one stabbed his girlfriend and was found by the Military Police digging a hole to bury her out in the camping ranges. She survived and he is, as they say in the Army, “Making little rocks out of big rocks” at a federal hard labor prison. These were people who could not bring themselves to adapt to the overbearing environment we spent way too long in.

But most of us kept our sanity. We did it by gaming the system. We would find ways to avoid marching in formation, even though it was an absolute requirement. We would go on road trips instead of subjecting ourselves to the busy work that the military calls “TAC Detail”. We would spend days sleeping in the barracks instead of painting curbs without getting caught. We became experts of the military bureaucracy and policy and used it time and time again to our advantage. And very few of us were ever caught. I actually had a copy of the Army’s dress code in my pocket at all times in case I was accused of not complying with them. Several “drill instructors” were amazed to learn that the regulations actually forbid polishing boots until they are shiny. Shiny is bad in battle.

It is not that we were bad soldiers. We were what I would consider the elite among us, the highly admired “skaters” and “shammers” who got out of anything we didn’t want to do, but we were all top notch students. We were doing the job that the Army required us to do, which was learning SATCOM to the best of our ability. The rest was just pointless BS that we made a game of avoiding. It wasn’t policy, it was harassment.

To highlight just how blatant and skilled we became at using the rules to our advantage, we may be the only platoon in the history of the Army to take a full PT test (the physical fitness test that is used as a Billy-club by many drill sergeants because it can determine whether or not you pass the school you are in, get a promotion to a higher rank, or even stay in the military) in full formation. The PT test is serious business as anyone who is or has been in the Army knows.

We had already completed our year of school (over 2000 classroom hours) and we were waiting to go to our first real duty stations. We were being held for a week until Christmas Exodus started (the two week Christmas holidays during which the regular military all but shuts down) and the order came down from our company commander that everyone must pass a PT test before leaving. We had just passed one the week before and were all completely fed up with the training base and the drill sergeants. The final PT test was just adding to the insult. As we saw it, we had paid our dues and this was just harassment. So we came up with a last-minute plan as we walked over to the PT field.

Since the PT test is such an important part of the Army, it is highly regulated. There are rules which state that “drill instructors” cannot force anyone to exercise the morning before a PT test, which means no “Drop and give me twenty pushups!” or any other type of physical reprimands. We also knew that once the PT test started, the drill sergeants could not interfere with us in any way. They could not insult us, obstruct us, or otherwise affect our ability to do our best on the PT test. Further regulations stated that they could not force us to do any physically strenuous activity for four hours after the PT test. It was carte blanch for us to behave badly and get away with it.

When the time came for our run, we all got into formation on the track. This was just downright weird because a PT test run is supposed to be an individual activity that shows an individual’s best running time. They knew the rules though and could not interfere. All they could do is read the rules script and blow the start whistle. And cringe. Because, being so close to Christmas, our platoon leader, genius that he was, started calling cadence. This is also off the wall in a PT test. But the best part was what the cadence consisted of. Not “Airborne Rangers rolling down the strip!”, but “Rudolf the red-nosed reindeer”. The drill sergeants’ faces all turned beet red. They knew they had been beaten at their own game.

“All of the other Reindeer”
“Reindeer!”
“Used to laugh and call him names!”
“Like Pinocchio!”…

It is a tradition on Army bases for the base commander (usually a General) to have a “Commander’s Run” on the morning just prior to Christmas Exodus. This is a big event because every platoon on base (or thousands of soldiers) all participate, representing their particular company. It really is an awe-inspiring thing to see thousands of fit, proud US Army soldiers running in a formation that stretches over a mile around the parade field. And the traditional cadences they sing with all of the gusto they can muster, trying to out-sing everyone else is incredible to behold, their regimental flags fluttering ahead of each company. But as the formation rounded the end of the parade field, they started falling silent and then slowly gave way to chuckles, then uproarious laughter. There on the test track was a small formation taking a PT test singing Rudolf the Red Nosed Reindeer. In cadence. Very loudly. And at the edge of the track were five drill sergeants with extremely red faces staring holes into the ground as their commanders, one by one, rounded the corner and realized what was happening. The General was not impressed. It was one of the greatest moments of my military service.

My point in telling this story is to illustrate what I have already stated. Smart people will not be forced to do things that are pointless. If they are, chances are very good that it will not turn out as you expected. Developers are smart people, many of whom consider configuration management as pointless, and who can blame them? Their experience has taught them that it never makes much difference in the end, it just adds to their workload.

So how is Integrity Control any different? If developers are such a smart bunch, what makes you thing that you can pull the wool over their eyes by changing the name of a process that they already perceive as pointless? Will they not just once again cheat the system and go back to doing things the way they think they should be done?

If that were all I was advocating, then yes, they would. It would be no different that calling a reprimand a "counseling session". Changing the name doesn't change the content. What is needed is a change of perspective, a new analysis of what we are really trying to accomplish by managing changes, configuration, and dependencies in a project. In the end, the questions that need to be answered are:

1. How does this make the developer's life better?
2. How does this increase value to the customer or client?
3. Is the tradeoff (effort vs. gain) worth it?

It may seem odd that the first question is focused on the developer. After all, he is paid to do the client's work. Why should his life being made better be of high importance on implementing something that is, in the end, supposed to benefit the client through a cleaner development process? The answer to that is obvious to me: “If the developer ain't happy, ain't nobody happy.”

We can debate the wisdom of catering to the demands of the developer and whether or not their paycheck (or lack of it) should be incentive enough to do as they are told, but as I already pointed out, developers are smart. This is a good thing, you need smart developers. But being smart, if they don't believe in what you are implementing, it will be doomed to fail. They will inflict a death by a thousand invisible cuts that are impossible to track down but lethal nonetheless, and it will be the project managers standing red-faced as they do the equivalent of singing a goofy Christmas carol during a project status meeting. Developers must be on board with the plan to protect a project's integrity or it is doomed to fail. More about how to do this soon.

The second question is "How does this increase value to the customer?" The customer is paying the bills. The customer wants everything now for nothing. Adding additional layers of work to a project translates to more billable time and higher cost. What is the client getting for this additional expense? A report each week that says "We are doing good"? How will controlling project integrity translate to value to the client? It must either result in less expense or better results to be accepted. In the end the client is interested in having bug-free software that does what they asked for it to do with the least amount of money expended. Therefore, to show value for a process it must add value by either saving time (billable hours) or by making the end product more reliable, easier to use, or implement more features than could otherwise be implemented.

The last question is the sum of the first two. It the tradeoff worth it? If we expend this extra effort, dedicate ourselves to yet another methodology, and do all of the things needed to make it work, will the time saved by having fewer bugs and easier to troubleshoot environments be worth the additional expense of keeping it up? Will the client see enough improvement in the product or the cost of obtaining that product to make it worthwhile for them to sign off on it?

Over the next few articles I will begin exploring these aspects of the real goal of project integrity control and how, by changing our views of what the reasons are for doing it, we can start to formulate a system that really works for the developer, the project, and the client and gets us closer to the goal of functional, bug-free software with a minimum of rework. I hope you will stick around for it.

M@