Why creating software really is like building cars
I would like to start by admitting that I am cursed. (SPOILER WARNING: if you are considering hiring me, please stop reading now!) I am an albatross to companies that hire me. It is not that I sabotage them or fail to do my job, it is just that I have an uncanny ability to take jobs for companies that are starting to unravel. For whatever reasons, I have had a really bad run of projects in my career as a developer. It would be difficult to go into detail without naming particular companies that most of people in this industry would recognize or may have worked for, but suffice it to say that I have witnessed not only the crumbling and collapse of many large projects firsthand, but I have also been around when the pink slips started going out and servers and network equipment were boxed up and sold off and the office spaces put up for rent. Thorough all of these tragic projects I have been baffled by the inevitable decline, unraveling, then total collapse of projects under their own weight.
Certainly a lot of the problems I have witnessed have been poor management in the form of lack of market research, grossly underestimating required capital, reckless spending, questionable accounting practices, too heavy reliance on new technologies, and misreading of the bones. In some of the cases, the project was dragged down as part of a larger collapse of the entire company and it would be inaccurate to blame those failures on the project itself, although I am quite sure that if the companies' problems hadn't killed them, the projects themselves would have self-destructed further down the road. I know that some people reading this are puzzled by my eternal pessimistic view of software projects because they have had the extreme fortune of working on projects that were well planned, well managed, and held to the high standards that any complex project should be. To these people, I say "count your blessings!"
This particularly bad run of luck with projects has produced a feeling that the software development world is a wild west "free for all" and that even the largest companies seem to be incapable of managing a software project to the same level of professionalism that they would manage other large projects. This is what has led me to make the much criticized comparison of software engineering projects to other engineering projects. I have received a lot of comments that it is a brainless comparison because building software isn't the same as building airplanes, cars, or bridges. There is some truth to those criticisms, but I fear the folks making them are missing my point. My point isn't that they aren't different, or course they are! My point is that we seem to be making little or no progress towards stabilization of software projects.
The main argument I have seen for why everyday software is so buggy is the one of costs and market forces. The argument goes "yes, you could have solid software. NASA has shown that. But the price of such software would be astronomical (pun intended) and nobody would be able to afford it." This seems like a legitimate argument on the surface, but I think it is flawed for several reasons.
First, software is a weird creature when treated as a product. It is generally sold the same way computers or keyboards or cameras are...as a product. But in reality, it is more of a service. A group of developers has provided the service of codifying complex business or other types of processes into a set of rules that can be applied by anyone with general knowledge of the business and basic computer skills.
Don't believe me? Then think about this: what are you buying when you spend $499 for a piece of software (or 499K for that matter)? Surely you aren't paying that amount for the actual disc. Even if it has a printed manual (yes, they used to do that!), the actual value of a couple of CDs or a DVD and a book is somewhere below $10. Some will say that you are paying for the privilege of using software, not the media or the software itself, and they are closer to the truth. When you purchase software, you don't own it any more than you own the songs that you download from iTunes. You are purchasing the right to use it. But why would anyone part with their hard earned money for the right to use something without owning it unless they got value from using it? They wouldn't. It is a given that if someone purchases something, it has a value to them. In music that value is the pleasure given by hearing that "YMCA!" remix and with software that value is whatever it is that the software does, makes easier, or makes possible that otherwise wouldn't be. You are paying for the service that the company that produces it provided up front in automating some difficult task.
I am going down this path to make a point. The common argument I see is that NASA produces reliable software and it costs millions of dollars. Who could afford that level of reliability other than an agency with deep tax pockets? The common computer user certainly couldn't. But what this argument misses is that NASA is pretty much their only customer. By the same logic, if a car can only be made reliable by investing millions of dollars in research and development, then a company that builds just a single car (as in one physical car, not one model) would have to charge millions for it just to break even. By this argument which is commonly given to justify sloppy software, nobody but the most wealthy could afford a reliable car. But that is not the case because this is not what car manufacturers do. They invest millions in making a reliable design, that is true. But then they reproduce that design at a minute fraction of the original cost. That design, reproduced tens of thousands of times, is then shipped to dealers who can afford to charge $25K for that same car that took millions to design and produce and still make enough commission to take a week long vacation to the Bahamas. In other words, the cost of producing a reliable car CAN be compared to the cost of producing reliable software if both have a large customer base. It is a valid comparison.
Once this point is understood, it stands to reason that software should actually be even easier to produce both reliably and cheaply than cars or airplanes. After all, each car produced contains several thousand dollars worth of raw materials and requires an appreciable amount of labor to produce. In addition to that, there are shipping, storage, and depreciation costs to consider. Software, on the other hand, is an all up-front prospect. Once the original, fully operational version of it is produced, the cost of producing another copy is negligible. So is the cost of producing the next copy, and the next, and the next million. The only raw materials needed are a couple of CDs or, more commonly, bandwidth on a network for providing downloadable copies of the software.
In this way software is more like a book. The original takes the author many months to prepare, but once it is complete, little or no further work is needed to regenerate it. If it is an eBook, no effort at all is required. Another download is just another opportunity to make money from the original. The risk in this model is that the author will spend months of his life writing a book that nobody will pay for. If this turns out to be the case and the author has figured his time as worth a certain amount per hour, then it is all in vain and he has lost a lot of time and income. But if he suddenly got a buyer for the first copy of his book from an individual, he would have to charge tens of thousands of dollars to that one customer to just break even. We all know that isn't the way it works though. A book provides value, but not that much value! The market will support a certain price level and that is what is charged for a book. If enough books are sold, the author makes money. If not, it is a wash and he takes some level of loss. This is a system that has worked for many years and has rewarded authors who provide relevant, quality work with a proportional reward and rewarded authors who produce poor quality work (in respect to value to customers) with no reward.
My assertion is that reliable software CAN be produced at a market sustainable price, similar to the example of writing a book. I know that I will get some arguments that somehow comparing creating software to writing books is invalid because (whatever), but that is the problem with comparisons. They are ALWAYS invalid at some level, otherwise they would just be restatements.
The problems I have seen seem to come in with custom development projects. They usually fall in the category of "one customer who must bear all of the cost". These projects lack the economy of scale that mass produced products enjoy and are like NASA and FAA projects in this respect. One client must bear the full cost and the argument can be made that, faced with the enormous cost of producing extremely high quality software, most companies settle for "good enough" software for a much lower cost. There is a lot of truth in this argument and I believe it is valid. But the flaw here is the assumption that the only way to produce quality software is by dramatically increasing cost. The implication is that clients put up with sloppy work because they can't afford high quality work. What this ignores is that there are many ways of controlling both cost and quality. In my next article, I will talk about some of the main causes of high cost and poor quality on software projects. See you then.
M@
Certainly a lot of the problems I have witnessed have been poor management in the form of lack of market research, grossly underestimating required capital, reckless spending, questionable accounting practices, too heavy reliance on new technologies, and misreading of the bones. In some of the cases, the project was dragged down as part of a larger collapse of the entire company and it would be inaccurate to blame those failures on the project itself, although I am quite sure that if the companies' problems hadn't killed them, the projects themselves would have self-destructed further down the road. I know that some people reading this are puzzled by my eternal pessimistic view of software projects because they have had the extreme fortune of working on projects that were well planned, well managed, and held to the high standards that any complex project should be. To these people, I say "count your blessings!"
This particularly bad run of luck with projects has produced a feeling that the software development world is a wild west "free for all" and that even the largest companies seem to be incapable of managing a software project to the same level of professionalism that they would manage other large projects. This is what has led me to make the much criticized comparison of software engineering projects to other engineering projects. I have received a lot of comments that it is a brainless comparison because building software isn't the same as building airplanes, cars, or bridges. There is some truth to those criticisms, but I fear the folks making them are missing my point. My point isn't that they aren't different, or course they are! My point is that we seem to be making little or no progress towards stabilization of software projects.
The main argument I have seen for why everyday software is so buggy is the one of costs and market forces. The argument goes "yes, you could have solid software. NASA has shown that. But the price of such software would be astronomical (pun intended) and nobody would be able to afford it." This seems like a legitimate argument on the surface, but I think it is flawed for several reasons.
First, software is a weird creature when treated as a product. It is generally sold the same way computers or keyboards or cameras are...as a product. But in reality, it is more of a service. A group of developers has provided the service of codifying complex business or other types of processes into a set of rules that can be applied by anyone with general knowledge of the business and basic computer skills.
Don't believe me? Then think about this: what are you buying when you spend $499 for a piece of software (or 499K for that matter)? Surely you aren't paying that amount for the actual disc. Even if it has a printed manual (yes, they used to do that!), the actual value of a couple of CDs or a DVD and a book is somewhere below $10. Some will say that you are paying for the privilege of using software, not the media or the software itself, and they are closer to the truth. When you purchase software, you don't own it any more than you own the songs that you download from iTunes. You are purchasing the right to use it. But why would anyone part with their hard earned money for the right to use something without owning it unless they got value from using it? They wouldn't. It is a given that if someone purchases something, it has a value to them. In music that value is the pleasure given by hearing that "YMCA!" remix and with software that value is whatever it is that the software does, makes easier, or makes possible that otherwise wouldn't be. You are paying for the service that the company that produces it provided up front in automating some difficult task.
I am going down this path to make a point. The common argument I see is that NASA produces reliable software and it costs millions of dollars. Who could afford that level of reliability other than an agency with deep tax pockets? The common computer user certainly couldn't. But what this argument misses is that NASA is pretty much their only customer. By the same logic, if a car can only be made reliable by investing millions of dollars in research and development, then a company that builds just a single car (as in one physical car, not one model) would have to charge millions for it just to break even. By this argument which is commonly given to justify sloppy software, nobody but the most wealthy could afford a reliable car. But that is not the case because this is not what car manufacturers do. They invest millions in making a reliable design, that is true. But then they reproduce that design at a minute fraction of the original cost. That design, reproduced tens of thousands of times, is then shipped to dealers who can afford to charge $25K for that same car that took millions to design and produce and still make enough commission to take a week long vacation to the Bahamas. In other words, the cost of producing a reliable car CAN be compared to the cost of producing reliable software if both have a large customer base. It is a valid comparison.
Once this point is understood, it stands to reason that software should actually be even easier to produce both reliably and cheaply than cars or airplanes. After all, each car produced contains several thousand dollars worth of raw materials and requires an appreciable amount of labor to produce. In addition to that, there are shipping, storage, and depreciation costs to consider. Software, on the other hand, is an all up-front prospect. Once the original, fully operational version of it is produced, the cost of producing another copy is negligible. So is the cost of producing the next copy, and the next, and the next million. The only raw materials needed are a couple of CDs or, more commonly, bandwidth on a network for providing downloadable copies of the software.
In this way software is more like a book. The original takes the author many months to prepare, but once it is complete, little or no further work is needed to regenerate it. If it is an eBook, no effort at all is required. Another download is just another opportunity to make money from the original. The risk in this model is that the author will spend months of his life writing a book that nobody will pay for. If this turns out to be the case and the author has figured his time as worth a certain amount per hour, then it is all in vain and he has lost a lot of time and income. But if he suddenly got a buyer for the first copy of his book from an individual, he would have to charge tens of thousands of dollars to that one customer to just break even. We all know that isn't the way it works though. A book provides value, but not that much value! The market will support a certain price level and that is what is charged for a book. If enough books are sold, the author makes money. If not, it is a wash and he takes some level of loss. This is a system that has worked for many years and has rewarded authors who provide relevant, quality work with a proportional reward and rewarded authors who produce poor quality work (in respect to value to customers) with no reward.
My assertion is that reliable software CAN be produced at a market sustainable price, similar to the example of writing a book. I know that I will get some arguments that somehow comparing creating software to writing books is invalid because (whatever), but that is the problem with comparisons. They are ALWAYS invalid at some level, otherwise they would just be restatements.
The problems I have seen seem to come in with custom development projects. They usually fall in the category of "one customer who must bear all of the cost". These projects lack the economy of scale that mass produced products enjoy and are like NASA and FAA projects in this respect. One client must bear the full cost and the argument can be made that, faced with the enormous cost of producing extremely high quality software, most companies settle for "good enough" software for a much lower cost. There is a lot of truth in this argument and I believe it is valid. But the flaw here is the assumption that the only way to produce quality software is by dramatically increasing cost. The implication is that clients put up with sloppy work because they can't afford high quality work. What this ignores is that there are many ways of controlling both cost and quality. In my next article, I will talk about some of the main causes of high cost and poor quality on software projects. See you then.
M@
[1] Comments:
I totally agree with you. The good news is that we will eventually come up with bits and peaces that will make almost any business process solvable with generic components. Like with cars. Not many companies have desire to design & build their own cars, they will usually settle for the one closest to their needs. Sure, this will radically reduce our industry but maybe that's a good thing because 90% of programmers are not well qualified for their job anyway.
Post a Comment
<< Home