Monday, July 27, 2009

Simple Software Cost Measurement

This was written at the same time as Averse to Change.

I’ve been jealous of mechanical or electrical engineers who can point at the direct material cost of their design, that concrete dollar amount, and say, that’s how much it costs. They gloat about redesigns that result in the same functionality but with a reduction in direct material cost. They rarely even have to talk about the cost in engineering time that it took to get to that cost reduction. Once I realized what the true value of software was, I realized that we too could use a simple measure like DM cost: lines of code. Not like the horribly misguided productivity measurements of old where more lines of code was better. When you realize that the ability to quickly make changes is priority number one, you see that less lines of code is better. I like this because it’s a lot like DM cost, where it’s easy to count and less is more, and even just a little less is usually worth the effort. In the physical world, that small reduction in DM cost is multiplied by volume. If a part is purchased in large amounts, a small reduction in cost equals a large savings. In software, even saving a few less lines of code can be multiplied by how many times the design may need to be changed and how much easier it will be to make those changes. A small reduction in lines of code equals a big reduction in total cost.

Wednesday, July 22, 2009

Averse to Change

This was written about a year ago and left unpublished because it talked about people I was working with. I like the people in the examples, despite my disagreements about they way they work, and I didn't want to hurt anyone's feelings, so I left it hidden on my hard drive. After a conversation a couple days ago at my new job about the apparent recent death of software engineering, I was reminded of this essay that I had written. I think it needs to see the light of day, so hopefully with the passing of time since the examples took place nobody will get upset. There are certainly no hard feelings on my side because I am far from perfect myself. OK, on to the essay.

Software (and the engineering that goes into crafting it) is a very misunderstood thing, especially at hardware focused shops. It is misunderstood by management, but that’s old news. I’m afraid it’s misunderstood by many of the engineers, even software engineers, as well. This leads to a lot of unnecessary difficulty and awkwardness in the software design and engineering process. Let me describe two scenarios from my experience that resulted from this kind of misunderstanding.


Bob is a programmer. He actually started his engineering life as some other kind of engineer. It doesn’t really matter, it could be mechanical, electrical, chemical, you name it, but somewhere along the line he got into programming and now that’s what he does. He seems to do solid work and is a smart guy. You notice, however that he gets more and more stressed as projects get closer to the end. Any new feature or new requirement or even a discussion of the possibility of making a change to the original design really sets him off. His face gets red and his volume increases at meetings. Everyone else gets uncomfortable. He fights every late-project code request tooth and nail but he usually manages to pull through in the end without too many late-project regressions in his code…

Then the day comes that you take a good look at his work. Maybe he takes over a another module that you worked on and you review his code after he’s been working on it a few months. As you look at what he’s done to your carefully crafted design you are aghast. Requirements have changed and some redesign was needed, but that’s not what Bob has done. No, rather than remove large chunks of code that should now be obsolete, he has bent over backwards to use the existing code as much as possible. Functions have been re-purposed and their names no longer make any sense. Variables names are now nonsensical. Some sections of code have been “removed” with prodigious use of ”#if 0…#endif,” and other sections of code have been routed around with conditionals, but nothing has actually been deleted or renamed. Some new functions have been added and lots of tweaking of parameters has been done in order to fit a new interface onto and over the old one. Hard-coded “special-case” code seems to be the rule rather than the exception. Copy-n-paste is not foreign to Bob’s repertoire either. The whole thing is a Frankenstein’s mess of duplicate code, spaghetti code, dead code, and just plain wrong code all mixed together. When you talk to him about his approach to this problem he excitedly tells you how little change he had to make to get the new requirements working. The only thing you get excited about is that you finally understand why it was so hard for him to make changes late in the game and why regressions arose why he did so. Bob is missing something fundamental about software engineering.


Fred is a programmer. He also has an interest in tools, and since nobody else expressed any interest he got the job as tools guy. Everyone else on the team loves that he talks to the IT department for them and that they never have to deal with IT directly. Still, there is some grumbling that they are using a 6-year old version of Linux on their development machines. Come to think of it, none of their other tools have been updated in quite a while. Haven’t there been some new features and bug fixes in that time? Since they are targeting an embedded system it doesn’t really matter which version of Linux they develop on, but they could at least use an update on emacs or vim, and getting help writing scripts in Python 2.2 is getting harder and harder. A few of them have started to compile newer versions of various tools like that on their own, but finding the right libraries for Redhat 8 is pretty tough.

Pressing management, someone finds out that he’s been told to upgrade the team’s tools, but he hasn’t made much progress. Talking to him he says that there just hasn’t been enough time for him to do it. How hard can it be, one asks? Get a spare workstation, slap the latest version of Linux on it, and then just make sure our cross-compiler still runs, right? Oh no, he explains. You see, we use ctags 5.2.3, which came with Red Hat 8, and we have to Verify that ctags 5.5.4 works the same on the new version of Red Hat. There’s also gftp, which people have gotten used to, which might not work the same when we upgrade. Fred also hasn’t had time to verify the new versions of ssh, python, bash, nedit, vim, emacs, and so, and so on. These open source people keep updating their software willy nilly, it’s hard to keep up, says Fred. One leaves this conversation with a clear understanding that Fred pays amazing attention to detail and that he has also erected an insurmountable barrier to changing the team’s tools.


I really had a hard time deciphering what it was with these people until I realized that they both had the same basic problem. They are totally averse to change. They are completely paralyzed by the thought of changing their carefully crafted systems, whether it be the code they’ve written, code someone else has written, or an assemblage of code that makes up a development toolset. Once they have something working, they are appalled by the idea of having to change it. This is unfortunate because it is a complete and utter misunderstanding of the true value that software has in a system.

When you design a system these days, software is always a given. I believe most assume it’s a cost thing. You could design hardware to do what the software is doing in most devices, but it would cost so much more. If that’s the case, why have any specialized hardware at all? Why not have a few general purpose processors and implement all of your features in software? Wouldn’t that be the cheapest way to do it? Hmm, well, those general purpose processors can be expensive, especially to get them fast enough to do things at the same speed that specialized hardware can do them. Sometimes many times more expensive than the cost of a specialized ASIC.

If that's the case, then why not craft your whole system out of specialized hardware? You can make it do things really fast, and with some good design effort and high enough volumes specialized ASICs can be very cheap. Very cheap, that is, until you find a bug in the ASIC, or realize there’s another feature you need. Then you find out how expensive it is to re-spin a chip, and the later in the project you have to do that, the more expensive it is both in terms of the money already spent manufacturing buggy parts, and in the money it will cost to make new masks and delay your project while the new version of the ASIC is in the fab. And will you have it all correct after only one re-spin? There’s a lot of risk there. That’s why hardware designers are averse to change; they should be! It’s expensive, so get the requirements right the first time, darn it, and you better not come late to them with a change in requirements.

Real Reason We Do Software

This is where the real value of software comes into play. It’s not that it’s intrinsically cheaper than hardware, it’s that it’s cheaper to change than hardware. The real value of software in a system is its malleability. It’s flexibility. The ability to make a late change to a feature, to fix a bug found close to shipping date, or to run a quick experiment is by far the main benefit of software. As software engineers, the best service we can provide to a project is to make changes to our code as quickly as possible. Reluctance and resistance to change our code and our software tools is a hold-over from hardware design and is the complete opposite of what we should be doing.

The ability to change and adapt is what should drive all of our design decisions when we craft software. The simpler a design, the more modular, and the less lines of code the better. The better tools we have to edit, refactor, branch, merge, and otherwise change our code the better. The faster we can confidently make changes to our code, the more value our software will add to a project. If we complain and resist changes to our code at every turn, if we ever consider ourselves done with our software, we are killing our contribution to the product. If we burden a project with cumbersome software change request procedures, we lessen the value of our software. The ability to quickly and easily be changed is software's core contribution to a system.

Wednesday, July 8, 2009

If You Have To Run Windows

If you have been spoiled by the developer heaven that is a well running Linux machine and you have to run windoze, this is what I’ve found that begins to almost make it bearable.

  • You must first open file explorer (windows-e) and do Tools->Folder Options->View and then check, “Display the contents of system folders,” and uncheck, “Hide extensions for known file types.” Also, View->Details is recommended.
  • Right-click on the question mark down near the bottom-right of your screen, left of the clock and all the icons (in the Taskbar), and chose “Hide the language bar” (or whatever the annoying thing is).
  • Right-click on the Taskbar itself and check “Show Quick Launch” and uncheck “Hide inactive icons”
  • On any command windows (cmd.exe, cygwin, etc.), fix the font. Click on the icon in the upper-left corner of the window and choose “Properties.” Click on the “Font” tab, and choose “Lucida Console.” You can make the window bigger with the “Layout” tab too, if you need.
  • Get a nice desktop background image
  • swap caps and ctrl. Scroll down a bit to find the registry script. And acutally, if you just make caps a ctrl, and leave ctrl as ctrl, other people that sit down at your machine won’t be so confused.

Now, install the following programs (and don’t allow any of them add an icon to your desktop, a Quick Launch icon is OK if you really use it a lot):

  • Firefox
  • txmouse.exe. Gives you X11-like copy-n-paste (even in cmd.exe windows) and focus-follows-mouse.
  • emacs w32. I went with the patched version this time, but I’ve used unpatched in the past haven’t noticed a big difference.
  • gimp
  • pidgin
  • VirtuaWin
  • cygwin. Select lots of goodies like gcc and openssh and whatever else your fingers type without thinking when you are at a command-line.
  • gnuwin32. Sometimes you end up at a command-line other than cygwin’s and your fingers still type things like ls, which, less, grep, and so forth. That’s what these are for.
  • locate32—fast file finder.
  • git for windows—if you need some solid open source revision control.
  • Console2. This is much nicer than the basic cmd.exe. You can resize it! It does transparency! and tabs! The neat trick is that you can make different types of tabs that each run their own shell, such as c:\cygwin\Cygwin.bat, or the git-bash shell. After you have made your tabs, create multiple shortcuts to start console 2, each specifying a different tab to start with using the -t option. You can give them the cygwin and git-bash icons to really polish things up.

And because I have Linux machines that my windoze machines must talk with:

  • winscp
  • putty. Not just for ssh, but for telnet (handy for embedded development), and works as a serial terminal too.
  • xming

For embedded software development:

  • pumpkin—hilarious looking and sounding, but functional, tftp server
  • Terra Term. For when putty doesn’t quite cut it as a serial terminal, and yes, the latest version is from 1998. That worried me also.