Atomic powered Project Integrity Management

When creating a system for keeping up with massive ongoing change, there are many things to consider and it is easy to become bogged down in the details of it all. This may explain why so few projects ever follow through with their configuration management plans. While it is obvious to many people involved with a project that some kind of change control and configuration structure is needed, it is not so obvious what that structure should look like.

As a software developer, I find that in situations where the problem I am solving is extremely complex it helps to approach it from different angles. Most things that are complicated at one level are fairly simple at another level. Biology and physics are chock full of great examples but the most readily understood one may be the atom. In the universe we occupy, there are untold levels of complexity and variation. Plants represent a different sort of complexity than mammals, and geographic formations and planetary orbits represent still an entirely different set. But when the problem is viewed from a lower level, similarities emerge that are not initially obvious. For example, all living things are made up of cells and require energy. They also all have a typical lifecycle. At an even lower level, the plants around us, the air we breathe, and the ground we walk on all have things in common, most notably that they are made of individual atoms of a finite number of elements.

That is great from a conceptual standpoint, but it doesn't do much for solving the problems faced with ever-changing projects much. It is too abstract to be useful in this situation, but it does illustrate a very useful technique for getting to the heart of a problem. What we need is an example that has already been figured out, tested and implemented to base our thinking on. Ideally it will be something less abstract, more physical and easier to picture. I think I have just the example needed.

Years ago transportation of goods was a very manual process. Farmers or shoemakers would gather their goods and haul them to a port where they were placed in wooden crates of various sizes then loaded by hand or with hoists onto waiting cargo ships. Those crates were then stacked as securely as possible in the holds of the ship where they were later hand-unpacked at the destination. Crates were then transferred to waiting trucks and boxcars where they were moved to their final destination where they were once again hand unloaded.

This was a process full of inefficiencies. In 1956 an American trucking firm owner named Malcom McLean purchased a used tanker ship and converted it to carry aluminum truck bodies in frames on its deck and started the company that would soon be known worldwide as Sea-Land. With the success of this idea came a revolution in cargo transportation. Seeing the need to standardize all containers so that they could be stacked securely and transferred from one mode of transportation to another, the standard shipping container was born. Today nearly all goods shipped across the oceans are packed in these standard containers. Now whether a container is filled with tomatoes, shoes, televisions, or computers, it can be handled and managed in exactly the same way with exactly the same equipment. Specialized trailers and rail cars allow containers to be transferred directly from the cargo hold or deck of a ship to an eighteen wheeler or rail car without ever being touched with human hands.

But packing everything in container only solves part of the problem. The real complexity of shipping still remains. Now that all containers look identical and could contain anything, a system is needed to identify and track the containers wherever they go. Some system must ensure that a container of oranges makes it to the grocery warehouse in Little Rock, AR on time.

The solution to this problem is the assignment of a unique identifier to each and every shipping container. These numbers are painted on the side and are used extensively to track where the container is going and where it has been. The container ID gives the ability to pull any amount of information about a container from any databases that use the identifier. By creating unique identifiers and maintaining databases of activities based on that ID, the problem of identifying and controlling each one is greatly simplified.

A typical shipping container

A cargo ship loaded with shipping containers

Shipping containers being unloaded.

So to summarize, the solution to the extreme complexity of packing, transporting, and tracking extremely variable products has been neatly solved by creating a standard package that all shipments are packed inside of. To facilitate tracking and routing, a unique number is assigned to each package. The combination of these two concepts, standard packaging and unique identification provide a powerful example of how extremely complex problems can be greatly simplified by abstraction. No matter what a container holds or where it is going, as far as the ship's captain, the crane operator, the railroad conductor, or the truck driver is concerned, it is a container with a certain destination.

Can you see where this example would be applicable to tracking components of a project?

In bookstores now! (not really)

As promised, this is the "dust cover summary" of what this blog is about. If this sounds interesting, maybe you will periodically come back and read the new installments. If not, then you are no worse off.

Project Integrity Management
Copyright © 2008, M@

January, 2005. After over four years and half a billion dollars, the FBI software system modernization project codenamed Project Trilogy was officially scrapped with no usable functionality to show. Every year, thousands of corporate software development projects around the world slip into the red, running over budget and over schedule. Many of them are scrapped and written off as complete losses while others are released incomplete and buggy. In the software development world, an on-time, on-budget, fully functional project is the exception, not the norm. Why?

Many will argue that software development is just too complex to be reliably managed. Software is typically not held to the same level of scrutiny that hardware projects are, giving much more responsibility to the developers to "do the right thing" in writing the code. In most cases, there aren't even established standards for how software is developed, tested, or deployed. Is it any wonder that problems are common?

But in "Project Integrity Management", the author makes the case that it is not the lack of standards or negligence of the developers that lead to software disasters; it is a lack of project integrity. This is a concept that creates a framework for managing the myriad of components that make up a software project. While most projects have tools and procedures in place for managing schedules, budgets, and source code, the large majority of them do not have anything in place to insure that each change to hardware, software, documentation, or procedures is done in a way that gives full visibility to all levels of the project.

It is the author's point of view that the vast majority of the problems that appear in a development project are the result of subtle changes and differences between the environments used to develop, test and deploy the applications. In a similar manner, the failure to effectively capture client expectations in requirements and manage changes to those requirements causes most of the resource sapping rework that projects seem doomed to go through.

When a project is under integrity management, each piece is accounted for and tracked as a small part of a larger unit. Just as individual wires, bolts, panels and other parts make up engines, dashboards and suspension on a car, each component of a integrity controlled project makes up the operating systems, databases, applications, and configurations. In the same way, just as engines, dashboards, and suspension make up a completed car, operating systems, databases, applications, and configuration make up environments.

This atomic approach to integrity accounting offers a manageable, easy to control framework that insures that the same code that worked in the development environment will work in the test environments and in production environments. It provides a way to insure that all parts are known in all environments, and provides a verifiable method of promotion of ALL changes (application code, hardware, operating system, configuration, etc.) from one environment to the next.

This sounds like a complicated thing to do on the surface. The variety and number of components in most projects is overwhelming, and once ALL changes are truly accounted for, they can come in at a furious pace. But by managing every item in a project as both a unique item and as part of a larger item, the entire project can be mapped and managed as a single, large but organized hierarchal structure. That is the power of Integrity Management!

-M@

Oh, the tedium!

I have received comments on my blog that indicate that some readers may doubt my genuineness on the topic of Project Integrity Control. Probably through some fault of my own, some people have gotten the impression that I am holding back to keep people coming back and that there may not even be anything to the whole topic. The fear is, apparently, that they will invest great amounts of their time reading installments of something that turns out to be impractical or just a rehash of old ideas. My posts have been called "tedious", which I can only take to mean "too long" or "too detailed". But unlike many things out there that are routinely blogged about (sports, politics, games), this is not a simple topic that can be covered in a few paragraphs and it takes a good understanding of the structure to realize the benefits of it. Maybe I am too detailed in my descriptions, but then again, I want to make sure that what I am explaining is what you are getting.

Still, I could be wrong and how would you know? You could just set a reminder in your datebook to come back in six months and take it all in at once, but if a person considers the 1-2 page articles tedious on their own, I doubt they will have the time or patience to get caught up all at once. Instead they will likely skip to the newest posts and then start arguing about things that they missed by not reading the previous ones. This is like complaining that an author is too tedious in a technical book and he should just summarize everything. Most all of them do this though. It is called the last chapter. Unfortunately without having read the previous 20 or 30 chapters, while it may be nice and summarized, you won't understand most of what it is talking about and will then have to read the "tedious" parts to gain that understanding.

I also realize that blogs aren't books and that the format that works for one may not neccssarily work for the other, but then again, I am not trying to get published...I already am published just by putting it in a blog. I am not trying to get traffic to support myself through banner ads and referrals. I have none. I am simply trying to explain something that I find very interesting in a way that will stick and make sense.

Still, I can sympathize with the point of view that maybe this is all hype and in the end, there will be nothing worthwhile. I have a funny name for that situation: life. But since I am comparing these articles to a traditional book (as opposed to a magazine article which typically wraps everything up in one installment), I will concede that books have something that helps with this problem that, so far, this blog does not: a dust cover summary.

I could argue the merits of doing such a thing on a blog...after all, the dust cover summary is a marketing tool and I have nothing here to market other than an idea which the reader can take or leave, but since I know that there is more to this series than dragging you on with the hope of some good information in the end I will shed some light on where I am going. I will write the equivalent of a dust cover summary for the concepts I am building here. This will give you the opportunity to decide whether or not I am nuts in one single reading instead of having to try to determine it over time. Then you can happily hang around and read them as I write them, or you can go somewhere less nutty and read other things with your time. Fair enough?

M@

Sharpening the Focus on ICIs

In order to show how mapping each item in a project to a ICI an make managing the project easier, it is necessary to define some type of structure that we can talk about. In this installment I will talk about the structure of an ICI and why it is so flexible.

As I have already pointed out, the first thing we know about an ICI is that it has a unique identifier. This can be in any form that you have a way of guaranteeing is unique. If you are a database-knowledgeable person, this would be equivalent to a unique key in a table. It must be entirely unique within a project. For the sake of simplicity, we will use an incrementing number for our examples, although in a real life situation I would opt for a Universally Unique Identifier (UUID) to guarantee uniqueness across projects in different databases. In addition to a unique identifier, what else do we need to manage an ICI? If you remember, we also talked about the need in IM to version everything. That implies that an ICI should have a version associated with it. It should. Each time the item that the ICI represents changes, the version should be incremented.

So far we have the following:

ID (unique identifier)
Version ID (incremented)

This is a good start, but that can't be all. I said that each ICI represents an item in the project. How do we know what that item is? Is it a .dll file? A configuration file? A SQL script? A user document? It seems that a good thing to know is what we are dealig with. Maybe I don't want to see all ICIs, only the ones that are user documents. Storing an indicator of what the underlying item is would be helpful in determining which ICIs I wanted to work with, so we should add a ItemType to our ICI.

So far, so good. We can uniquely identify this item, we can find out what version it is at, and we can get a general idea of what it is. But what if I want to actually get a list of the documents, not just a bunch of unique IDs? In this case, we would want to add a Source File value to it. But wait...what if I want to only see the user documents that are for the system administrator? Should I add an "intended audience" value also? That seems to make sense unless you consider that we could be talking about a .dll file instead of a document. Now "intended audience" makes no sense at all. We seem to be headed down a dead-end road.

If you take this discussion to its logical end, you will realize that everything that could possibly be tracked about ANY item in your project will need some place to put data. This littl ICI is quickly starting to turn into a massive data structure with potentially hundreds of data items. It is a database designer's nightmare! What to do?

If you think back to the last discussion, I talked a little about the powers of abstraction. By ignoring some things, or looking at them at a higher level, you can do some pretty amazing tricks that are not immediately obvious at the lower level. In some cases they are downright impossible. But if we reframe the problem, a solution will emerge soon enough. Lets pretend for a moment that we are not talking about a project, but a retail order. An order can have a single item, or several items. Come to think of it, an order can have an unlimited number of items. How it this normally handled? Skipping over the technical explanation, the answer is that at the order level, all items, no purchase what they are, are handled as "order items". Like ICIs, they can represent anything in inventory. With this approach, we don't have to worry about whether or not an individual item has a particular value because it is still an order item and can be treated like all others. While they may have some common pieces of information (stock number, quantity, price, etc.), the products that these pieces of information represent are really not very much alike.

Coming back to ICIs, we identified that there are several things that are common between all ICIs, but potentially many more that are not. The way to handle this seemingly impossible situation is to abstract this never-ending pile of possible data items down to a single thing: a Property.

Let's take a look at what that does for us. For our purposes, a property is defined as "a piece of information relevant to an ICI". It is obvious that we must be able to associate multiple properties to a single ICI, but once we come to that conclusion, the vexing problem that we were facing melts away. Now a .dll file can have certain properties (file name, size, install folder, etc.) while a web server can have other properties (Manufacturer, model, processor(s), RAM, etc.). By breaking apart properties from the ICIs that they are associated with, we have solved the problem of how to anticipate all possible pieces of information to be tracked. The solution is that new properties are created as they are needed. This solution, however, leaves a problem to be solved. On one side of the room we have ICIs and on the other side of the room we have a big pile of properties. How do we know which properties go with which items? This is a fairly easy problem to solve logically. With retail orders, each item has an associated "order number". The order number is unique per order (just as an ICI id is unique per ICI). When an item is added to an order, a line item record is created that contains both the order number and the product code for the item ordered as well as other pieces of information. Now we have a link that ties our item ordered to a particular order. To reconstruct the order, all we have to do is retrieve all line items that match the order number.

This is similar to what is done with an ICI. As properties are added for an ICI, they store not only the property data, but also the unique ICI ID.

A simple puzzle. You have a room full of kids and no information about who they are. You don't even know their names. You also have a list of parent names. How do you figure out who each child's parent is? The answer is pretty obvious; you ask the kids. The point is, the kids know who their parents are so you don't need any other method of figuring it out. in fact, you don't even need to know anything about the child, you just need to ask what their mom or dad's name is and find them on the list.

Similarly, in the ICI scheme we are discussing it is very easy to figure out which property goes with which ICI because the property "knows" who their parent is just as the order items "know" which order they belong to due to the order number "link" that each one contains. This is referred to as a "linked list" in programming and is a common method for linking a single item to multiple sub-items.

Armed with our new technique of defining properties as they are needed and linking them to an ICI, we can now put the problem aside of figuring out what an ICI looks like. It is sufficient to say that an ICI has a unique ID and one or more linked properties. Problem solved! We have come full circle and in the end, decided that the only thing we need in an ICI is a unique identifier value. Everything else can be linked to it as we need to add information. We will leave the actual "how" of linking properties to ICIs for a later discussion. Just trust me that it can be and is done every day in software that you probably use.

There seems to be a axiom emerging as we continue to discuss the methods we can use to create simplicity from complexity and order from chaos. You will see this theme emerging time and time again as we progress because it is a very simple and powerful technique. What it boils down to is "If you don't know the answer, try changing the question." Not something you are likely to get away with in other parts of life unless you are a politician, lawyer, or philosopher, but it works in Integrity Management quite well.

Now that we are mentally at peace with our ability to forge ahead without having to know and account for everything we may encounter along the way we can start looking at ways we can leverage this newfound power to make the job of creating software more reliable and easier.

ICI Basic Training

Last time I talked about how every part of a project must be uniquely identifiable and explained how the Integrity Control Item (ICI) can be used to perform this task. What I didn't really explain was what an ICI looks like, where it lives, and what it can do other than be unique. While being unique is a full time job for many celebrities, it is not extremely useful on its own. If we are going to go through the trouble of giving something a name, it should do a little more than sitting around being unique to earn it! The ICI does. It should provide some practical and useful information of functionality.

First, let’s talk about the structure of an ICI. The question of "what does an ICI look like?" is comparable to the age-old mystery of "how long is a piece of string?” In the case of the string, it is as long as it is, and in the case of the ICI, it looks like it looks. So much for practical, useful information! The reason for giving this apparently useless description (other than wanting to use the string question) is to drive home the point that the ICI is a concept, not a thing. Like a color, it can be useful but it is hard to pick one up and put it in your pocket (no, that is a crayon you are thinking of...only school kids call them "colors"!). This is because as useful as colors are, they are not things. We can speak of them (I have on my blue shirt today) and use them to convey information (the blue lights in my rear view mirror); colors just cannot exist on their own. Instead, they describe other things that already exist. We can even group things logically by color that have no other common properties (make a list of the blue items you see in the room). In many ways, this is what ICIs do. They provide a convenient way to talk about and manage things that may not be very alike otherwise.

In the software world, as many other disciplines, we say that the ICI functions as an abstraction of the various pieces that make up a project. It gives us a way to perform the same actions on very different things that would otherwise be difficult or impossible. For example, your project may contain a configuration file and a packaged database system. Other than being file-based, these items have very little in common. If they were people they would stand on other sides of the room at parties. But what if the hypothetical people are both huge Dolphins fans? This is a common thing that they have in common that they can both relate to, so while the conversation is on politics, they don't talk. DBMS is a Republican and the configuration file is a Democrat. When the conversation turns to movies, still they have no common ground. DBMS is an action movie fan while CF only likes documentaries. But when the subject turns to football, suddenly they are both interested and you can't get them to shut up. That is until DBMS's wife, Web Server, whispers that he should talk to some people of his own grain for a while...then they again drift off to opposite sides of the room. Pretty dramatic, huh?

The purpose of an ICI is to provide that common ground between all of the different pieces of your project. Like Army soldiers' proclamation of "I'm not black or white, I am green" and the even broader "soldier fist" motto, ICIs ignore the differences between the pieces and leverage the commonalities. No matter what a soldier's job is whether field cook, generator repairman, or helicopter pilot, you can assume several things about him without even knowing him. First, he knows how to fire and field strip an M-16. He knows how to throw a grenade without killing himself, and he will put his life on the line to defend his flag. These are things that are common to everyone who wears the Army uniform. They may have different taste in music, different jobs, different religions, and different political views, but when an officer barks out "attention!” they will all snap to attention in the exact same manner.

In the same way the military trains each of its members with common skills and responsibilities, ICIs define things about any item in a project that can apply to every single one, no matter what it is.

This may sound like some kind of shell game of pretending that things are the same when they aren't, but it isn't at all. The key to understanding how to implement ICIs is to understand that nobody is born with the reflex to snap to attention when an officer yells "Atten-hun!" (Anyone who thinks they yell "ten-hut" has never actually been in the Army). This is an artificial behavior that is introduced in their training to allow leaders to control them as a single unit when needed. Using the techniques collectively know as "drill", a leader can move 1000 soldiers with the same effort it takes to move 10. This is significant because anyone who has ever been in a crowd of 1000 or more people trying to move from one place to another will have the word "mob" in their mind. With so many different people with so many different ideas about how to get from point A to point B, the resulting chaos is typically extremely inefficient. By introducing some artificial behaviors into this crowd, they can be treated as a single unit, moving in step and in perfectly straight columns, turning, stopping and even saluting in perfect order. This is the power of abstracted behaviors.

It is important to note that introducing these standardized behaviors does not in any way diminish the individual's ability to do their job or move about on their own when coordinated movement is not required. The same soldier who marches in perfect synchronization with 999 of his fellow troops can, in the next moment, take up a lone position and effectively protect it without any further orders from his commander. He can even be left at this post for days and will perform his job as required with no further intervention. But when he rejoins his platoon and the "FALL IN!" order is given, he will be just as efficient in moving with the others as he was before. Those who have participated in marching bands will recognize the parallels between ICIs and marching drill as well. Each member has their own part to play (literally), but it takes a common set of drill steps and maneuvers to control the overall musical production.

The purpose of an ICI then is, on an abstract level, the same as the purpose of marching drill. It defines a set of characteristics that can be relied on no matter what the underlying item it represents is. For those familiar with object oriented programming, an ICI could be considered an instance of an IC class. They all have the same structure but represent different distinct items.

Once we have established what the structure of our ICI is, we can then "map" it to any item in our project whether it is hardware, software, documentation, or even other abstract things such as routine maintenance tasks and procedures.

Now that you understand the concept of what an ICI is and how it is used, next time I will talk about how bringing all project under the collective control of ICIs can impressively simplify some of the most complex tasks that face projects as they move through the development, testing and deployment phases. Hope you will join me.

-M@

A huge floating mass of metal or a collection of units?

In my last article I talked about the need for a unique identifier for each Integrity Control Item (ICI). I am going to expand on that concept this time around. If you end up following this series all the way through, when you look back you will see that the IM is so critical to getting control of your project that without it, there isn't much chance of pulling it off successfully. It will, one day, seem obvious.

The Integrity Control Item is a very basic unit. It is used to represent the smallest practical unit in an Integrity Controlled project. What is meant by "smallest practical unit"? The way I like to think of it is to compare pieces to atoms. That probably sounds a little off the wall, but having a high interest in Physics, that comparison just makes the most sense. Let me explain.

Everything in the universe is made of something. Well, in theory anyway. I have a personal theory that once physicists followed the route of existence down from "rock" to "mineral" to "element" to "molecule" to "atom" to "electrons, protons and neutrons" to "quarks" to "sub quarks and gluons" we are in grave danger. I can just imagine the fateful day that some physicist in a high energy lab somewhere makes the discovery that the entire universe and everything in it is made up of absolutely nothing. Suddenly everything disappears, leaving the limitless vacuum that everyone seems to agree once existed. Sounds like a Douglas Adams story.

Like physical matter, everything in a project, no matter how complex that project is, is made up of something. The closer to that something you get, the less the things differ. A ship is a very complex system, but if you divide it into hull, deck and superstructure, it can be logically separated. Designating decks, compartments, on down to individual pieces of equipment give an even more detailed view. Now we are standing on the deck of a ship looking at a single bolt. Is that the smallest practical part of a ship? I would say it is one of many. In fact, if you think about what we just did, something interesting is going on. Call it Conservation of Complexity.

We started out imagining a ship. It is a single item. If you saw it coming into port, you would not tell someone "wow, look at that massive collection of steel plates, bolts, pipes, wires and paint!" You would more likely say "wow, look at that huge ship! How in the world does that thing float anyway?" But that is a different discussion. The point is that the ship you see pulling into port represents a single item, but an extremely complex one. Since the human mind is only capable of focusing on one or two items at a time, we see a ship, not all of the individual parts. But something weird happens. Once you start looking at it as a collection of parts, the overall number of items skyrockets. Instead of a ship, you have a hull, deck, and superstructure. And within that superstructure you have windows, doorways, passages, rooms, radar dishes, radio antennas, etc. If you were to examine a single room, you would find innumerable parts make it: steel plates, beams, insulation, pipes, wires switches, valves, and fixtures. If you were to examine the sink in a room you would find that it contains a myriad of parts as well: brackets, pipes, valves, screws, gaskets, etc. Examining the drain pipe would reveal individual sections of curved pipe to make up the trap, nuts to hold them together, gaskets to prevent leaks, a ball valve to close the drain, linkage to the drain stopper lever, and on and on. But have you noticed that although the number of things in the ship has grown exponentially by examining them at close range, their complexity has decreased. A single room is much less complex than a complete deck. A sink is simpler than a room, and a drain is simpler than the whole sink.

Yes, that is obvious, so what? Why do I care?

Because ships offer a very good model for software projects. Like projects, they are extremely complex, they require many people with many different skills working together in a very organized manner to slide into the water after having a bottle of bubbly broken on its bow. The effort required to plan the designs, procure the materials, control the order that things are built, make sure that steam lines are not run next to fuel tanks, and the millions of other details that go into building a ship is at least as great as the most complex software project. Yet ships do, generally get built and most of them don't just sink when they are launched. This fact offers hope to those of us who feel that software is just too complex to do correctly. So are ships, but they get built.

So how do shipbuilders handle the overwhelming complexity of their trade? In short, they choose simplicity over component size. They design things from the top down, but they keep working it until they get down to a "lowest unit" level. They may be building a hull, but they do it in small job-sized pieces. If a group of engineers got together one Monday morning to start developing a new ship design and for whatever reason decided that they would start with the sinks in the living compartments and work up to the overall design, how long do you think that would take? How correct do you think the designs would be? I think they would be very wrong indeed. Unfortunately since the design is from the lowest level to the highest, they would not discover that they had neglected to consider where the water for the sink was going to come from, or possibly the drain emptied into the fresh water tank below. In order to fix these problems, they would have to tear out huge sections of their work and constantly rework it. Sounding familiar?

Not that these things don't happen. My brother works for a major US shipbuilder as a welder. He told me an amazing story one day that illustrates that sometimes even shipbuilders get it wrong. He was working diligently to finish up a welding project on a wall insider the belly of a new ship. He noticed as he was working that another person was setting up a cutting torch and then just waiting. "Am I in your way?" my brother finally asked him. "Nope, go ahead.' was the answer. Half an hour later the same guy was still standing there waiting. Finally it began to bug my brother and he asked again, "Are you sure I'm not in your way?". "Nope, I can't start until you finish and get your work signed off" was the reply. "Oh, OK. What are you going to be doing?" Imagine the dismay when my brother heard his answer "Cut out that wall you are welding. We have a design change and that wall is moving." This may seem outrageous...OK, it IS outrageous, but if you really think about it, you have probably seen some comparable things happen in software projects.

Circling back around to Integrity Control Items, software projects are very complex, but one way of getting a handle on that complexity is to take a tip from shipbuilders and track each part individually. This is what IIs do for us in Integrity Management. They define a single unit (for example, a .dll file or a file server) and allow it to be treated, for the most part, exactly like any other unit. This greatly simplifies the day to day things that have to be tracked, but it also creates a large amount of data to be tracked. So, as was hinted at earlier, we have traded complex large items for simple, numerous small items. Like the cans of corn and bottle of salad dressing with UPC codes mentioned in an earlier article, we can now operate on these very different items with the exact same logic.

Hopefully this has gotten you thinking a little about how complexity can be conquered by decomposing the complex items into many simple items. Next time I will dive a little further into how this is actually done in the real world.

-M@

Digging in to Project Integrity Management

What is "Project Integrity Management"? How is it different from Configuration Management? How does it make it any easier to prevent the dreaded project meltdown? I'm glad you asked.

As mentioned in a previous article, Configuration Management attempts to control elements of a project, specifically controlling and managing the constant change of information associated with the project. The amount of information to be tracked can be overwhelming, and the task soon becomes too complex to be performed manually. This leads most projects to seek a "toolset" to manage their configuration for them. While many commercial and some open source toolsets exist today, they all seem to be solving different problems. This is most likely due to the variations in different people's understanding of what Configuration Management consists of.

What many projects call CM is actually not configuration management at all, but simply source code control. For this task, there are a number of software packages available that provide varying degrees of functionality and frustration. But source code control is not configuration management. I would even go so far as to say that it is not even part of the configuration management process, but I will save that argument for a later time when I feel like starting a flame war.

At this point you may be thinking "this guy is either crazy or high"! I assure you that I am neither. Read on.

In the interest of avoiding splitting hairs on what Configuration Management is or is not, I have opted to create a new term altogether that will represent the concepts that I feel are the ones that need to be defined in order to get a handle on the mess that many of us know as the "development project". This term is Project Integrity Management and although it is similar in some ways to Configuration Management, the two are not interchangeable, at least not with the definitions I have seen of CM. So from here on out we will be speaking of IM instead of CM unless I am referring specifically to a Configuration Management concept.

First of all, IM is not source code control. This has been done many times over and, although I don't think that the approaches that are currently being used are great, I have nothing to add to that debate. It isn't that I don't think source control is important, I just think that the process is well understood and we should focus on some of the more vexing problems that have not been so thoroughly studied and developed for.

At the highest level, IM consists of the two following areas:

Versioning
Environment Management

These two concepts represent everything needed to completely track and control a project's variables. As you may have guessed, there are many lower levels to these concepts which we will explore, but first I would like to describe what is meant by these two terms.

Versioning is the most important concept in IM. Without versioning, the rest is impossible. In IM, everything is versioned. If it is not versionable, we are not interested in it because the definition of "versionable" is "possible to change". If something is impossible to change throughout the life of a project, then why would we care about it? It is in an eternally known state and tracking it would be silly. What would we track? The catch in this definition is the very important word "possible" is a pretty wide-open one. For example, you may consider that the operating system that is being developed against is not going to change during the project, but you would likely be wrong. Service packs change operating systems all of the time. Servers can crash and be replaced with different machines. Documents can be edited, corrected, and revised. The upshot of this is that pretty much everything associated with a development project can change. Nothing is "set in stone". What that means in IM is that everything must be versioned. Yes, that includes operating systems, servers, and documents. If it can change and it is part of a project (i.e. part of a deliverable), it must be versioned. We will explore the finer points of versioning soon, but for now this short discussion will serve our purposes.

The other high level concept in IM is Environment Management. This may or may not be a new term to you; I didn't make this one up! But I am going to heavily redefine it to suite IM's purposes. What I am referring when I mention Environment Management is the control of all versioned items in each project environment. Examples of environments in this context are "development", "test", "training", and "production". However, as you will find out, this is not to say that there is only one development or test environment. In most projects there are several. The IM approach to Environment Management is one of "Just In Time" environment creation and destruction. Environments are like Krispy Kreme doughnuts. As soon as they are created, they start going stale. I will also provide a detailed approach to implementing this JIT environment control plan.

Those are the high level concepts. On their own they are not very useful though. In fact, at this point, they may not even be comprehendible to any meaningful degree. But if you could see the picture in my head (which I hope to bring you around to doing), you would see that not only is it comprehendible, but very elegant and flexible. Now that we have gotten the mile-high view, I want to jump to the other end of the spectrum and give you a peek into how this can be pulled off. But first, a quick trip to the store.

Some of you older folks may remember a time when grocery stores put price stickers on everything. Each can of green beans or salad dressing had a white or orange sticker on it with a price. No barcode, just a simple number like "$1.29". When you checked out, there was a cashier who could (brace yourself) actually type numbers into a cash register. In fact, she could typically type them very fast with one hand and move the groceries to the bagging area with the other.

One day I noticed that some things that I bought at the grocery store had a new "built in" tag with a lot of lines on it. It seemed completely useless to me because the only thing on it besides the row of lines was a seemingly random series of numbers. I eventually learned that these were called "UPC" or Universal Product Code tags. Since the tags were implemented long before most grocery stores received and installed the cash registers that could read these tags, they seemed rather pointless to most people. But soon all of that changed. Because each product carried a unique code, and that code was the same for that product whether it was on a store shelf in Akron Ohio or Pascagoula, Mississippi, products could be paid for, inventoried, and ordered all using the UPC number. In fact, those processes eventually became completely automated. Modern checkouts at large chain stores such as Wal-Mart or Target read the UPC code, look up the price, subtract a single item of the matching type from inventory, and if the supply is getting low, automatically order new stock. There are currently some experimental stores that constantly monitor supply and demand and adjust the price of each item in the store based on these real-time statistics!

It is difficult to overstate the far reaching impact the simple UPC has had on the retail industry. It has enabled stores with millions of items in stock to instantly know how many of each item is on the shelf, how many are on order, and how many were sold in the past hour.

That is all very interesting, but what has it go to do with what we are talking about? I am about to tell you!

Before the days of UPC, the task of pricing, inventorying, and ordering stock was very time consuming. Since the cashier only had the price that was paid for an item to enter (not what the item was), there was no way to know how much the inventory of a particular item had been depleted other than manually counting items or, more commonly, "eyeball ordering". A department manager ordered new stock when the stock on the shelf looked low. In many ways, the problems they faced had a lot in common with the problems we now face on software projects. We have a huge number of individual pieces which must be assembled in just the right way to work. To make things worse, they are always changing. It would be as if there were gremlins in the store that constantly rearranged the shelves when nobody was looking and mixed different items up. Actually, I think those really did exist. They were called kids. I know because I mixed up a few shelves myself. With this overwhelming amount of information to track and with many, many items changing all the time, something is needed to get the kind of control on project pieces in a similar way UPC codes allowed stores to get control of inventory. We need a UPC for our project pieces.

Enter the Integrity Control Item. This is a concept that treats anything that is versioned (i.e. everything we care about) as a unique item. Similar to the way a can of Green Giant whole kernel corn has a code that is unique throughout the food supply system, our ICI is unique throughout our project. Actually, properly implemented it is globally unique even between projects, but more on that later. For now it is sufficient to say that a ICI is unique in a given project. In our examples, I will just use a numeric value like "324398675" to represent an ICI.

So what does that do for us? Great. Now instead of worrying about the Oracle Client drivers, I can worry about keeping up ICI #324398675 and its thirty eight thousand brothers. Yea, buddy...that sure made my life easier! But wait...there's more!

If the only thing we got out of an ICI was a hard to remember number it would be no better than a cell phone, but that is not the case. When a grocery store orders something with a UPC code, when the order comes in, it has additional information with the UPC code. For example, it may have the manufacturer, the product name, the quantity, and the country of origin. You don't need to know the country of origin to order the corn, you just need to know the UPC code, but if you need to know what that UPC represents, a large amount of information about it is available in the inventory control system. Similarly, if we want to automatically check that an ICI is present in a particular release, we don't need to know what it is or what its characteristics are ahead of time, those can be looked up when they are needed or not looked up when they are not. There are many, many other advantages to tracking everything in your project with an ICI which will become more obvious as we dig deeper into this process, but for now I will let you ponder these ideas and try to shoot holes in them.

Next time I will discuss how versioning works with an ICI. I hope you come back to read about it.

-M@

Why creating software really is like building cars

I would like to start by admitting that I am cursed. (SPOILER WARNING: if you are considering hiring me, please stop reading now!) I am an albatross to companies that hire me. It is not that I sabotage them or fail to do my job, it is just that I have an uncanny ability to take jobs for companies that are starting to unravel. For whatever reasons, I have had a really bad run of projects in my career as a developer. It would be difficult to go into detail without naming particular companies that most of people in this industry would recognize or may have worked for, but suffice it to say that I have witnessed not only the crumbling and collapse of many large projects firsthand, but I have also been around when the pink slips started going out and servers and network equipment were boxed up and sold off and the office spaces put up for rent. Thorough all of these tragic projects I have been baffled by the inevitable decline, unraveling, then total collapse of projects under their own weight.

Certainly a lot of the problems I have witnessed have been poor management in the form of lack of market research, grossly underestimating required capital, reckless spending, questionable accounting practices, too heavy reliance on new technologies, and misreading of the bones. In some of the cases, the project was dragged down as part of a larger collapse of the entire company and it would be inaccurate to blame those failures on the project itself, although I am quite sure that if the companies' problems hadn't killed them, the projects themselves would have self-destructed further down the road. I know that some people reading this are puzzled by my eternal pessimistic view of software projects because they have had the extreme fortune of working on projects that were well planned, well managed, and held to the high standards that any complex project should be. To these people, I say "count your blessings!"

This particularly bad run of luck with projects has produced a feeling that the software development world is a wild west "free for all" and that even the largest companies seem to be incapable of managing a software project to the same level of professionalism that they would manage other large projects. This is what has led me to make the much criticized comparison of software engineering projects to other engineering projects. I have received a lot of comments that it is a brainless comparison because building software isn't the same as building airplanes, cars, or bridges. There is some truth to those criticisms, but I fear the folks making them are missing my point. My point isn't that they aren't different, or course they are! My point is that we seem to be making little or no progress towards stabilization of software projects.

The main argument I have seen for why everyday software is so buggy is the one of costs and market forces. The argument goes "yes, you could have solid software. NASA has shown that. But the price of such software would be astronomical (pun intended) and nobody would be able to afford it." This seems like a legitimate argument on the surface, but I think it is flawed for several reasons.

First, software is a weird creature when treated as a product. It is generally sold the same way computers or keyboards or cameras are...as a product. But in reality, it is more of a service. A group of developers has provided the service of codifying complex business or other types of processes into a set of rules that can be applied by anyone with general knowledge of the business and basic computer skills.

Don't believe me? Then think about this: what are you buying when you spend $499 for a piece of software (or 499K for that matter)? Surely you aren't paying that amount for the actual disc. Even if it has a printed manual (yes, they used to do that!), the actual value of a couple of CDs or a DVD and a book is somewhere below $10. Some will say that you are paying for the privilege of using software, not the media or the software itself, and they are closer to the truth. When you purchase software, you don't own it any more than you own the songs that you download from iTunes. You are purchasing the right to use it. But why would anyone part with their hard earned money for the right to use something without owning it unless they got value from using it? They wouldn't. It is a given that if someone purchases something, it has a value to them. In music that value is the pleasure given by hearing that "YMCA!" remix and with software that value is whatever it is that the software does, makes easier, or makes possible that otherwise wouldn't be. You are paying for the service that the company that produces it provided up front in automating some difficult task.

I am going down this path to make a point. The common argument I see is that NASA produces reliable software and it costs millions of dollars. Who could afford that level of reliability other than an agency with deep tax pockets? The common computer user certainly couldn't. But what this argument misses is that NASA is pretty much their only customer. By the same logic, if a car can only be made reliable by investing millions of dollars in research and development, then a company that builds just a single car (as in one physical car, not one model) would have to charge millions for it just to break even. By this argument which is commonly given to justify sloppy software, nobody but the most wealthy could afford a reliable car. But that is not the case because this is not what car manufacturers do. They invest millions in making a reliable design, that is true. But then they reproduce that design at a minute fraction of the original cost. That design, reproduced tens of thousands of times, is then shipped to dealers who can afford to charge $25K for that same car that took millions to design and produce and still make enough commission to take a week long vacation to the Bahamas. In other words, the cost of producing a reliable car CAN be compared to the cost of producing reliable software if both have a large customer base. It is a valid comparison.

Once this point is understood, it stands to reason that software should actually be even easier to produce both reliably and cheaply than cars or airplanes. After all, each car produced contains several thousand dollars worth of raw materials and requires an appreciable amount of labor to produce. In addition to that, there are shipping, storage, and depreciation costs to consider. Software, on the other hand, is an all up-front prospect. Once the original, fully operational version of it is produced, the cost of producing another copy is negligible. So is the cost of producing the next copy, and the next, and the next million. The only raw materials needed are a couple of CDs or, more commonly, bandwidth on a network for providing downloadable copies of the software.

In this way software is more like a book. The original takes the author many months to prepare, but once it is complete, little or no further work is needed to regenerate it. If it is an eBook, no effort at all is required. Another download is just another opportunity to make money from the original. The risk in this model is that the author will spend months of his life writing a book that nobody will pay for. If this turns out to be the case and the author has figured his time as worth a certain amount per hour, then it is all in vain and he has lost a lot of time and income. But if he suddenly got a buyer for the first copy of his book from an individual, he would have to charge tens of thousands of dollars to that one customer to just break even. We all know that isn't the way it works though. A book provides value, but not that much value! The market will support a certain price level and that is what is charged for a book. If enough books are sold, the author makes money. If not, it is a wash and he takes some level of loss. This is a system that has worked for many years and has rewarded authors who provide relevant, quality work with a proportional reward and rewarded authors who produce poor quality work (in respect to value to customers) with no reward.

My assertion is that reliable software CAN be produced at a market sustainable price, similar to the example of writing a book. I know that I will get some arguments that somehow comparing creating software to writing books is invalid because (whatever), but that is the problem with comparisons. They are ALWAYS invalid at some level, otherwise they would just be restatements.

The problems I have seen seem to come in with custom development projects. They usually fall in the category of "one customer who must bear all of the cost". These projects lack the economy of scale that mass produced products enjoy and are like NASA and FAA projects in this respect. One client must bear the full cost and the argument can be made that, faced with the enormous cost of producing extremely high quality software, most companies settle for "good enough" software for a much lower cost. There is a lot of truth in this argument and I believe it is valid. But the flaw here is the assumption that the only way to produce quality software is by dramatically increasing cost. The implication is that clients put up with sloppy work because they can't afford high quality work. What this ignores is that there are many ways of controlling both cost and quality. In my next article, I will talk about some of the main causes of high cost and poor quality on software projects. See you then.

M@

Your first big challenge...getting the developers on board.

Your In my last article I put for the theory that the most important thing to do when considering a new methodology, especially something as far reaching as CM (or as I have renamed it, Integrity Management - I can do that, its my article), is make sure you have the support of your developers. They are smart folks and it is much better to have them on your side than against you. So how do you do that? How do you get a bunch of "I'll do it my way, thank you" hermits to agree to do extra work for the good of the project? The most obvious way to me is to show them that it really is for the good of the project. Most developers really do want their projects see the light of day and very few I have met actually want a project to fail. But you have to keep in mind that if they have been doing this kind of work very long they have been fed many lines about a new framework, a new development methodology, or a new tool that is supposed to make their job easier and the end result more reliable. They are going to be skeptical, and that is a good thing. I personally don't want a bunch of gullible developers who jump on the next newest new thing that comes out. I like to see some caution when it comes to changing things that can make or break development project.

I once worked for a small company that prided itself as one of the only "Microsoft Premier Partner" shops in the area. They stuck Microsoft posters all over the place and hung the Premier Partner plaques (the clear plastic ones with Bill Gates "signature" on them) in the front lobby. They plastered their web site with Microsoft logos and "awards" (MVP, MCDS, MCP, etc.). It was pathetic.
Needless to say, when Microsoft came out with a new technology (or a repackaged existing one), our company president couldn't contain his ecstacy. He would have the admins immediately load it on a server and start "taking it for a spin". For the next two weeks we developers would get a constant barrage of emails with code snippets in them and subjects like "LDAP is so easy with this new _______" or "S-WEEET..Exchange interactivity via XML SOAP Protocol", or whatever. You get the idea.

Inevitably within a few weeks he would have satisfied himself that this new technology wsa top notch and ready for prime time, even if Microsoft hadn't (Or as he would yell from his office at regular intervals, "Rock On!"). He would then have all of the developers drop the work we were doing on real customer requested features and bugs and start recoding the entire app for whatever this new technology was. I was coding in .Net in late 2001 and early 2002, back when even Microsoft considered it Alpha. Not in "hello world" projects, but in the company's main product, a very large home healthcare system.

To keep this environment going, you can't afford to have any devlopers who were cautious or thought for themselves. They all had to toe the line and follow the Microsoft lead in everything. We added remote messaging to the application to report application errors. Why? Because these things called "sinks" in the Enterprise Application Architecture Library COULD. No other reason needed. He also had us require the user to enter a description of the error and what they were doing when it occurred. Apparently (as I predicted) there is a particularly troublesome area of the application commonly referred to as "asdf", because that is what most of the replies contained. Some of them were more colorful and described what the computer (and by extension, we developers) could do with the error message. Little was left to the imagination on those. Did I mention that once we got the entire application converted to .Net (by then it was in Beta), we started converting the whole UI to Infragistics .Net Ultragrids? Those were beta as well and oh, boy what a bucket of fun that was!
Looking back, I see now that what that shop needed was a few skeptical developers who would stand up and say "that's dumb and I am not going to do it". At the time I was so happy to be back at work after a six month dry spell that I wasn't going to do it. But I should have.

After this experience, I am naturally cautious of any developer that responds to an announcement that we are going to "re-architect", "re-design", or "re-implement" anything, much less the way we control the changes that happen in an application development cycle with anything other than objection. So the first rule is: If nobody objects, be careful; they are either new graduates who haven't learned yet, or they don't plan on doing it anyway. They are just trying to get you to go away so they can get back to coding.

If they are good at what they do, you will get instant pushback in the form of "Oh, so this is going to fix all of our problems just like “Agile” did? Or was it “CMMI” that was going to do that? No, wait, I think it was “RUP” or maybe it was ___________!

Don’t panic; this is a good sign because if they are skeptical and you bring them around, you will know that they are sincere about doing it, and if you have been led down the primrose path by me, they will let you know where the gaps are.

As I mentioned in my last article, the way to the developer's heart by making his job easier, but only if it appeals to his built-in logic processor. Less "stupid _ _ _ _" and more development power is the key. A good developer can spot the holes in any logic from five miles up, so you have to know that what you are advocating is sound. Once you appeal to their sense of logic ("hey, you know, that makes a lot of sense!") and you demonstrate how it will make their job easier by taking the bull off of them, you will win them over.

I know this because I am, at heart, a developer and I have never been convinced of the value of CM until recently. In fact, it took me approaching CM the same way I would any other project to get it broken down to the point that it made sense.

The next article will begin to describe a system of project integrity control that is based on very sound, very flexible and very simple logic. We will start at the most basic level of tracking and expand it to show how it can be applied to any project, no matter how complex it is. I hope you come back to read it.

M@