One of my favorite things about software engineering is the short feedback loop. It is one of very few professions that allow you to postualte, implement, and test a solution to a problem within hours or even minutes. And if it doesn't work, you simply try something else. The cost of trying new things seems pretty low when all you need to do tweak some text files.
And when you do solve a problem it's a rush. When your solution is elegant it's almost intoxicating. We love solving problems elegantly through:
- Efficient data structures and smart algorithms
- Clever abstraction
- Clean code, with little or no duplicated logic
Elegant solutions are where I think engineers tend to think of themselves as craftsmen. However, elegant solutions are not always good solutions. Elegant solutions are often the beginnings of technical debt.
Elegance is not an end. This is obvious. But I think the line between high quality solutions and elegant solutions is often blurred. It's often hard to recognize in the moment that we're going too far, that we need to stop being so clever.
My Lesson in Elegance
When I was a consultant, I worked on a large project for a private equity fund-of-funds client, replacing their propietary investment transaction system. The old system was written in Delphi, and was starting to fail badly. A lot of the application relied on database triggers and stored procedures to do the heavy lifting of calculations and business logic. The Windows application would deadlock on large calculations like the year end valuations of funds. Almost all other technology in their stack was running on the .NET framework as an internal web application or service, which were much easier to administer and deploy.
Accountants spent a large amount time doing very manual calculations to break out transactions by pro-rata (by ownership percentage). In particular their clients invest annually in their funds, those funds make investments in external partnerships. However the cash flows and returns are often sent at different levels in this hiearchy. Sometimes through other investment companies or comingled funds. Another time sync was moving data between different systems.
So in short, the general goals were to:
- Move the technology to the .NET web stack
- Make the user inteface fast and similar to Excel
- Integrate with other services to speed up data entry
At the onset of the project, my main focus was on these problems:
- The transactions could be entered and edited from the parent and child level of the transactions.
- How do we reconcile the dollars entered at different levels of the hierarchy? How do you maintain the pro-rata constraints?
- How do we deal with rounding error?
- How do we share this logic across all transactions?
- How do we deserialize and serialize this information from a database row?
- How do we validate that the data was entered correctly for different transactions?
I wrote a lot of code to make all of this work in what I felt at the time was a clever, elegant solution. The hierarchical nature of these transactions led me to using a genericized tree to hold an abstract class Flow, eg
Tree<TransactionFlow>. The calculations were all done by traversing the tree, either up or down depending on the way the data was entered. Each concrete implementation of the flow object would handle it's own process for validation and database object. I became obsessed with refactoring as much of the commonalities out as I could. And I wrote all of this before we built out the rest of our phases.
But as the project dragged on and as I was getting ready to leave my job, it became more and more clear that I had saddled the team with technical debt. I was trying hard to build something maintainable and elegant, but the deep levels of abstraction were hard to understand and follow. The changing requirements made a lot of the code obsolete or difficult to maintain. At times it was important for a user to just override the calculation workflow or the validations altogether. Stock distributions just didn't work the way cash draws and distributions did, neither did valuations. What was at first a simple algorithm to distribute dollars from parents to children, became a lot more complicated.
What went wrong?
At the time I started on this project, I had just finished reading the book Code Complete 2. This is a fantastic book and it's been very impactful on my career. However, the lessons from this book and others like it need to be administered pragmatically. I was refactoring for the sake of refactoring. Cleaning up code before it was really a problem.
My favorite quote about this is from Christopher Miles' post, Java Doesn't Need to be So Bad
I'm advocating an approach where you're mindful of these design patterns but you keep them in the background. When you have working code that is starting to get messy, or when someone on your team has been banging their head on a problem for way too long, that's when you take out the patterns book. Because at that point you have a pretty good idea of what the problem actually is. You even have a solution that's mostly working and that solution can be re-factored to fit the pattern that seems most appropriate.
Producing high quality code is good, but balancing this goal with the ability to easily understand and maintain the solution is more important. My solution could have been a lot easier to understand if I was willing to let things be a little more repetitive at the beginning. The commonalities would have been much easier to factor out during the project than way at the beginning.
Lack of Feedback
When you are writing elegant code, you think you're making rational decisions. I thought my abstraction of the transaction model made a lot of sense and I knew that we had to build many more flavors of the same thing in the coming phases of the project. What I failed to recognize is that the requirements were more fluid. That the user might need more flexibility in how they worked with the software.
This is why getting regular feedback on your code is critical. If your colleagues can't understand your code with some basic explanation, that's a bad sign. This is also why it's important for multiple people to also work on the same code base.
Focus on Building for Today, not Tomorrow
Having focus is so important. It's one of the lessons that I have learned well while working for a startups in San Francisco. You can have a vision for the company, and your projects, but right now you need to solve today's problems, not tomorrow's. I was building for problems months in advance when they weren't yet fully undertstood. Even now, it's occasionally very tempting to build with the anticipation of problems in a month, but I have to stop myself. I don't have enough information to make those decisions.
Recognizing the Problem
It's pretty easy to get mad at the people who wrote over-engineered code that you have to maintain, chalking it up to incompetence or even malice. I'm guilty of this, too.
But the nuance here is that I didn't do this out of malice and I think of myself as a good engineer. I was trying to do the right thing. That's the irony of most technical debt. It's built with the best intentions.
It's easier to see this problem in others but not necessarily ourselves, as John Mathis points out:
.@ifandelse good architecture is when I do something well-abstracted and flexible. over-engineered is when someone else does the same thing— John Mathis (@johndmathis) December 14, 2012
Unfortunately, a lot of software engineering education and even our interview processes are built around these age old questions:
- How efficient is your code?
- What's the runtime of your algorithm?
- How can you make it faster?
- How clean is your code?
- Etc, etc, etc.
It's drilled into our heads that elegant code is a good thing, that it's the mark of a good software engineer.
While all of these concepts are important to understand, I think it's more important to know when these things really matter and when they don't. When to apply these principles and when not to. That's the mark of a great software engineer.