As an engineering manager of a small team, I sometimes find myself trying to articulate why a solution is too “hacky”. Hackiness is a nebulous concept in software engineering, but it presents itself with some tell-tale signs:
- Hacky solutions often have a much smaller time estimate than the original gut estimate from leadership
- Hacky solutions are often praised by hackers but strike vague fear into the hearts of architects
- Hacky solutions often procure watery, quasi-religious counter arguments about doing the right thing in engineering
First, let me clarify what I mean by “hacker” and “architect”. In software engineering there is a spectrum between hacker and architect. Hackers derive solutions from experimentation with technology and a deep understanding of its properties. Their solutions are less concerned with best practices than with subtle affordances provided by the underlying technology. Architects derive solutions from software engineering best practices. They will find or create technology to implement these best practices, ignoring any shortcuts or hidden boons from technological details. This short post offers a bit more detail, but the point is that they have a yin-yang relationship. Most engineers skew to one side but do not like to be labeled as either.
In big tech companies, hacking is often associated with being junior while architecture is associated with being senior. This creates a pattern where a junior engineer presents a design with some amount of hackiness, and they receive endless pushback from senior teammates. The interesting part is that the arguments are usually not over whether to hack. Instead, they are about whether the proposal is even hacky. The challenge is coming to agreement on a hackiness appraisal.
Describing hackiness is not easy. Engineers sense hackiness with an intuition that has developed over years of experience. Distilling that feeling into words can manifest as clumsy objections about why the solution won’t scale. These points often feel too hypothetical and distant from the problem at hand. Hacker types respond to these criticisms with explanations of why the solution can be adjusted for high scale later on. They discount the future suffering from rebuilding a live system. And because software engineering has a very collaborative culture, the discussion can continue for a long time.
Similar to how “hacker” can be used as an insult, architects have their own kryptonite: accusation of over-engineering. Over-engineering is a company killer, and it can be downright sinful. In its worst form, over-engineering is an indulgence by engineers who use company dollars to educate themselves in complex systems. As legendary technical manager Andy Grove points out, the tuition for learning from unguided mistakes is paid by the customer. Even without malice, over-engineering implies that an engineer cannot actually solve business challenges, relegating them to a career of ticket-wrangling without agency. A hacker can often deploy the over-engineering label to retort criticisms of a quick and dirty solution.
Another complicating factor is the bias of each side. The architect and the hacker share a desire to succeed, but they have different success criteria. The hacker often evaluates themselves based on shipping prodigious amounts of code and getting features into production quickly. The architect often prizes behemoth design documents and execution of long-term plans. These desires can cloud their judgment of which approach is best for the problem at hand.
If only we could find some criteria to assess hackiness and spare everyone from this cycle! Unfortunately, the haziness of the term is baked into its definition:
Hacky: 2. (of a piece of computer code) providing a clumsy or inelegant solution to a particular problem.
If we squint at this definition, we can see that the wiggle room lies in the particular problem. This suggests that having a better way to define a problem can result in a shared understanding of its acceptable hackiness. Too often, engineers point to a problem’s complexity as justification for a commensurately complex solution. Instead, we should focus on its consequences.
The complexity of a solution is proportional to the problem’s consequences.
For a given problem, there is often a set of solutions with varying complexity. All of the solutions technically solve the problem, but only one is most appropriate for the consequences. Focusing on these consequences shifts attention from what is best in a general sense to what is best in this instance.
Consider parallel parking. Parallel parking is one of the most difficult parts of driving. Unskilled drivers commonly bump the adjacent cars as they maneuver into a spot. But based on the number of scuffed up cars in my Brooklyn neighborhood, people don’t seem to mind the bumps too much. If they did, then more drivers would either pay for a garage or equip their car with a bumper protector. Given the low consequence of a bump here and there, it seems reasonable that we solve parallel parking with some basic training and a brief exam.
Now, imagine that a parked car could easily explode when bumped. All else equal (bear with me on ignoring secondary effects), parallel parking would be a lot scarier. Giving teenagers a quick test and sending them onto the roads would be egregiously careless. Instead, we would pursue a more complex solution. Some that come to mind are automatic parallel parking, parallel parking as a service, or subsidizing construction of more parking garages.
Second, consider securing an asset from theft. If the asset is my favorite hacky sack, I might hide it in a shoe box under my bed. Nobody really wants the hacky sack, and the chance of a home invasion is pretty low. Now suppose that the hacky sack is actually a sack full of diamonds. I would have to upgrade to a defense-in-depth strategy that would take the likes of an Ocean’s 11 heist to be compromised. And these protections still would not employ even more comprehensive measures like geosharding the diamonds, so it would be vulnerable to natural disasters or an evil sovereign state.
When going through this exercise with your own examples, it’s important to remember two caveats:
- This idea only applies to comparing solutions for a specific problem. What is hacky for one problem is over-engineering for another.
- The problem needs to have at least two solutions to compare anything. Unsolved problems like time travel or problems with only one known solution do not apply.
Assessing Consequence
So if we can assess a problem’s consequence, then we can use that as a proxy for its acceptable hackiness. Broadly, we can consider a few different axes for assessing a problem’s consequences: scale, cost, permanence, and risk to life. In my experience, engineers tend to focus on scale while discounting the other three.
Scale: The expected usage of the solution, usually related to user engagement over a time interval. For most engineering projects, this is the most obvious factor. The most common problem I have seen is underestimating scale, especially because the business usually cannot tolerate halting operations when the anticipated scale is exceeded.
Cost: The amount of time, money, or human resources to be invested in a solution. Engineers are often very distanced from a project’s costs, the bulk of which may actually be their own employment!
Permanence: The degree to which the solution cannot be modified after its initial delivery. Permanence can be subtle and overlooked. The solution may need to support old versions of itself for many years, especially if dictated by business contracts. Additionally, deploying a new version into the wild can take a very long time (i.e. patching client-side software).
More cynically, note that permanence may be underestimated because the time horizon is longer than the engineer’s tenure at the company! I believe that this does happen, but blame lies more on the incentive structure of the organization than on the employee. This tangent probably deserves its own post.
Risk To Life: The likelihood that the solution’s failure could damage life (human, animal, or even plant). Many engineering projects have very low risk to life, but tail risk must be carefully considered. For many solutions, even a single fatality can be disastrous for the business (see Peloton treadmill death or Tide Pod deaths).
These four factors can be used to anchor a discussion about acceptable hackiness. Instead of devolving into a holy war about the right way to be an engineer, arguments can be framed in terms of the problem’s consequence profile. I suspect that this line of argument may be especially effective for convincing junior engineers because they usually have very little experience working on high-consequence problems. Pointing out the vast difference in consequence between at-home or academic projects and the solution at hand can illustrate the need for more rigor.
By the same token, these factors provide justification for a small startup’s fast-paced decisions. If the solution has few users, low cost, can be easily rewritten, and does not endanger lives, then by all means hack away!
Conclusion
The lens of consequentialism offers an informative way to assess solutions. In particular, the amount of redundancy in a system is a textbook tradeoff between cost and reliability. If a crypto enthusiast says you should use a Coinbase wallet and Vitalik Buterin says you should have multiple wallets, they can actually both be right. It really depends on what is at stake.
Next time you need to walk the tightrope between hackiness and over-engineering, consider the consequences! Hopefully you’re working on something impactful enough that failure will affect someone somewhere, but you probably aren’t playing Squid Game.
Thanks to Stephen Malina and Corinne Hardy for a careful review of this post.