Averse to Change

This was written about a year ago and left unpublished because it talked about people I was working with. I like the people in the examples, despite my disagreements about they way they work, and I didn't want to hurt anyone's feelings, so I left it hidden on my hard drive. After a conversation a couple days ago at my new job about the apparent recent death of software engineering, I was reminded of this essay that I had written. I think it needs to see the light of day, so hopefully with the passing of time since the examples took place nobody will get upset. There are certainly no hard feelings on my side because I am far from perfect myself. OK, on to the essay.

Software (and the engineering that goes into crafting it) is a very misunderstood thing, especially at hardware focused shops. It is misunderstood by management, but that’s old news. I’m afraid it’s misunderstood by many of the engineers, even software engineers, as well. This leads to a lot of unnecessary difficulty and awkwardness in the software design and engineering process. Let me describe two scenarios from my experience that resulted from this kind of misunderstanding.

Bob

Bob is a programmer. He actually started his engineering life as some other kind of engineer. It doesn’t really matter, it could be mechanical, electrical, chemical, you name it, but somewhere along the line he got into programming and now that’s what he does. He seems to do solid work and is a smart guy. You notice, however that he gets more and more stressed as projects get closer to the end. Any new feature or new requirement or even a discussion of the possibility of making a change to the original design really sets him off. His face gets red and his volume increases at meetings. Everyone else gets uncomfortable. He fights every late-project code request tooth and nail but he usually manages to pull through in the end without too many late-project regressions in his code…

Then the day comes that you take a good look at his work. Maybe he takes over a another module that you worked on and you review his code after he’s been working on it a few months. As you look at what he’s done to your carefully crafted design you are aghast. Requirements have changed and some redesign was needed, but that’s not what Bob has done. No, rather than remove large chunks of code that should now be obsolete, he has bent over backwards to use the existing code as much as possible. Functions have been re-purposed and their names no longer make any sense. Variables names are now nonsensical. Some sections of code have been “removed” with prodigious use of ”#if 0…#endif,” and other sections of code have been routed around with conditionals, but nothing has actually been deleted or renamed. Some new functions have been added and lots of tweaking of parameters has been done in order to fit a new interface onto and over the old one. Hard-coded “special-case” code seems to be the rule rather than the exception. Copy-n-paste is not foreign to Bob’s repertoire either. The whole thing is a Frankenstein’s mess of duplicate code, spaghetti code, dead code, and just plain wrong code all mixed together. When you talk to him about his approach to this problem he excitedly tells you how little change he had to make to get the new requirements working. The only thing you get excited about is that you finally understand why it was so hard for him to make changes late in the game and why regressions arose why he did so. Bob is missing something fundamental about software engineering.

Fred

Fred is a programmer. He also has an interest in tools, and since nobody else expressed any interest he got the job as tools guy. Everyone else on the team loves that he talks to the IT department for them and that they never have to deal with IT directly. Still, there is some grumbling that they are using a 6-year old version of Linux on their development machines. Come to think of it, none of their other tools have been updated in quite a while. Haven’t there been some new features and bug fixes in that time? Since they are targeting an embedded system it doesn’t really matter which version of Linux they develop on, but they could at least use an update on emacs or vim, and getting help writing scripts in Python 2.2 is getting harder and harder. A few of them have started to compile newer versions of various tools like that on their own, but finding the right libraries for Redhat 8 is pretty tough.

Pressing management, someone finds out that he’s been told to upgrade the team’s tools, but he hasn’t made much progress. Talking to him he says that there just hasn’t been enough time for him to do it. How hard can it be, one asks? Get a spare workstation, slap the latest version of Linux on it, and then just make sure our cross-compiler still runs, right? Oh no, he explains. You see, we use ctags 5.2.3, which came with Red Hat 8, and we have to Verify that ctags 5.5.4 works the same on the new version of Red Hat. There’s also gftp, which people have gotten used to, which might not work the same when we upgrade. Fred also hasn’t had time to verify the new versions of ssh, python, bash, nedit, vim, emacs, and so, and so on. These open source people keep updating their software willy nilly, it’s hard to keep up, says Fred. One leaves this conversation with a clear understanding that Fred pays amazing attention to detail and that he has also erected an insurmountable barrier to changing the team’s tools.

Misunderstood

I really had a hard time deciphering what it was with these people until I realized that they both had the same basic problem. They are totally averse to change. They are completely paralyzed by the thought of changing their carefully crafted systems, whether it be the code they’ve written, code someone else has written, or an assemblage of code that makes up a development toolset. Once they have something working, they are appalled by the idea of having to change it. This is unfortunate because it is a complete and utter misunderstanding of the true value that software has in a system.

When you design a system these days, software is always a given. I believe most assume it’s a cost thing. You could design hardware to do what the software is doing in most devices, but it would cost so much more. If that’s the case, why have any specialized hardware at all? Why not have a few general purpose processors and implement all of your features in software? Wouldn’t that be the cheapest way to do it? Hmm, well, those general purpose processors can be expensive, especially to get them fast enough to do things at the same speed that specialized hardware can do them. Sometimes many times more expensive than the cost of a specialized ASIC.

If that's the case, then why not craft your whole system out of specialized hardware? You can make it do things really fast, and with some good design effort and high enough volumes specialized ASICs can be very cheap. Very cheap, that is, until you find a bug in the ASIC, or realize there’s another feature you need. Then you find out how expensive it is to re-spin a chip, and the later in the project you have to do that, the more expensive it is both in terms of the money already spent manufacturing buggy parts, and in the money it will cost to make new masks and delay your project while the new version of the ASIC is in the fab. And will you have it all correct after only one re-spin? There’s a lot of risk there. That’s why hardware designers are averse to change; they should be! It’s expensive, so get the requirements right the first time, darn it, and you better not come late to them with a change in requirements.

Real Reason We Do Software

This is where the real value of software comes into play. It’s not that it’s intrinsically cheaper than hardware, it’s that it’s cheaper to change than hardware. The real value of software in a system is its malleability. It’s flexibility. The ability to make a late change to a feature, to fix a bug found close to shipping date, or to run a quick experiment is by far the main benefit of software. As software engineers, the best service we can provide to a project is to make changes to our code as quickly as possible. Reluctance and resistance to change our code and our software tools is a hold-over from hardware design and is the complete opposite of what we should be doing.

The ability to change and adapt is what should drive all of our design decisions when we craft software. The simpler a design, the more modular, and the less lines of code the better. The better tools we have to edit, refactor, branch, merge, and otherwise change our code the better. The faster we can confidently make changes to our code, the more value our software will add to a project. If we complain and resist changes to our code at every turn, if we ever consider ourselves done with our software, we are killing our contribution to the product. If we burden a project with cumbersome software change request procedures, we lessen the value of our software. The ability to quickly and easily be changed is software's core contribution to a system.