From Software to Software Engineering: Abstractions

Recently I wrote down my thoughts about how Software Engineering, as a true engineering discipline, was really only just beginning. I also opined that, as Software Engineers, we have an opportunity to do something other engineering disciplines cannot: remove the gap between design and implementation.

How can Software Engineers make it so that what they design is, demonstrably, what has been implemented? And how can we do that without losing the speed and efficiency wins that working with AI coding models has provided? The answer is quite simple: abstraction.

Of course, the concept of abstraction is nothing new in the development of software. The question is whether powerful enough abstractions would allow Software Engineers to design systems with sufficient constraints and specifications? Recently, while discussing this question over dinner, the classic analogy of a car was brought up.

When you get into a car, sit behind the wheel, turn the key and pull out to whatever destination lies ahead of you, you are benefiting from a massive number of powerful abstractions. Turning a key or (increasingly often these days) pushing a button to start the engine abstracts away the process of bootstrapping the engine with a starter motor, adjusting the fuel/air mix as the engine warms up, timing the ignition to ensure efficient combustion, and literally dozens of other things that have to go right for your engine to be able to power your car. Similarly, the shifter, accelerator and brake pedals, steering wheel, and pretty much every other piece of a car that you, as a driver, touch are also abstractions.

The critical challenge is: if an engineer was working at the level of these abstractions, could they still create specifications sufficient to determine if all the implementation underlying these abstractions was correct? In other words, could an engineer design a car at the level of the user interface (all the parts that a user touches directly) and hand that design over to a team and tell them to “go build it”?

I would argue yes, but it may not be obvious to a non-engineer just what this means, or what it would take. For example, what happens when you depress the accelerator pedal in a car? If you said, “it goes forward,” then you’re not thinking like an engineer. An engineer would first ask, “Is the car turned on?”, followed by “Is the shifter in reverse, drive, or neutral?”, and “Is the brake pedal being depressed at the same time?”

In fact, not only is it possible to create a specification for the complete operation of a car based on these user interface abstractions alone, but doing so is a prerequisite to building the best car possible. To understand why, we need to delve briefly into the concept of a “leaky” abstraction.

What do we mean when we say that an abstraction leaks? Consider that abstractions exist so that behavior can be described at one level without concern for how the lower levels operate. If understanding the behavior of some element of an abstraction depends on understanding a detail of those lower levels, then it is a leaky abstraction. A good example is any moderately complicated electronic device. Typically, you can use your computer or tablet or smartphone without care or concern about how it was programmed, how those programs are executed, or how the processors, memory, and circuits connecting them are organized. But every time you encounter a problem with one of these devices and resort to the immortal advice of The IT Crowd, “Have you tried turning it off and on again?”, the abstraction is leaking.

The reason a power cycle can resolve issues is a direct result of how the state of a system propagates through the layers of implementation that the device abstraction exists to protect you from. When memory is not cleared, or a program does not return to a predictable state, and the only solution is to wipe everything out and start from the beginning again, you are suffering from a leaky abstraction.

Similarly, on older cars with manual chokes, which required the operator to consciously alter the fuel/air mix in order to get the engine running, the car-starting abstraction was leaky. Now, an engineer working on such a car could include manual choke settings in the specification, so the act of simply building a specification around a user interface is not sufficient to build powerful abstractions. Using a pure user interface abstraction to build a verifiable specification does, however, surface abstraction leaks.

That is, if you were to build a car with a manual choke and not include the choke settings in the specification of the ignition system, then you would not be able to say, conclusively, that the car is operating as designed by verification of the specification (as the car may fail to start with an incorrect choke setting while still adhering to the letter of the specification). On the other hand, including the choke setting in the specification surfaces, in dramatic fashion, where the abstraction is leaking. The complexity of the specification reflects, exactly, the complexity of the user interface. There is no hiding from it.

So how is it that car buyers tolerated, or were subjected to, leaky abstractions for decades? And how would specifying software at the level of the user interface be any more successful at creating better abstractions? This is where the gap between design and implementation comes back into our story. The engineer designing a car does not only have to account for the potentially leaky abstractions in their design, but also must account for manufacturing tolerances, variances in assembly, and the general reality that a design on paper will never be the same as the physical article. A sufficiently well specified car design will never be sufficient to ensure that a car actually runs. Physical testing and quality control processes are a necessity, and if both the design engineer and the quality control technician are so intimately familiar with the operation of a manual choke that they don’t even think about the need to specify its operation in the design or testing procedure, then it is completely understandable how such a leaky abstraction makes it through to the finished article.

On the other hand, the Software Engineer has a bit of a super power in this regard. If the design and implementation are one and the same, then a completely specified user interface is sufficient to guarantee a correct implementation. Similarly, any failure of implementation should be directly traceable back to a failure in the specification. If, on the other hand, software is specified in one system, and implemented by another, only to have the behavior of the implementation empirically tested by adherence to the standard, then Software Engineering finds itself caught in the same trap as every other engineering discipline. Every time you have developed a website and, instinctively when the site starts misbehaving, you reload the tab, you are effectively doing the same thing as the automobile assembly line quality control technician instinctively adjusting the manual choke when checking to see if a car starts.

Where does AI fit into such a vision of the future of Software Engineering? More on that next time…