Here’s an insightful paragraph from James Hague’s blog post Organization skills beat algorithmic wizardry:
When it comes to writing code, the number one most important skill is how to keep a tangle of features from collapsing under the weight of its own complexity. I’ve worked on large telecommunications systems, console games, blogging software, a bunch of personal tools, and very rarely is there some tricky data structure or algorithm that casts a looming shadow over everything else. But there’s always lots of state to keep track of, rearranging of values, handling special cases, and carefully working out how all the pieces of a system interact. To a great extent the act of coding is one of organization. Refactoring. Simplifying. Figuring out how to remove extraneous manipulations here and there.
Algorithmic wizardry is easier to teach and easier to blog about than organizational skill, so we teach and blog about it instead. A one-hour class, or a blog post, can showcase a clever algorithm. But how do you present a clever bit of organization? If you jump to the solution, it’s unimpressive. “Here’s something simple I came up with. It may not look like much, but trust me, it was really hard to realize this was all I needed to do.” Or worse, “Here’s a moderately complicated pile of code, but you should have seen how much more complicated it was before. At least now someone stands a shot of understanding it.” Ho hum. I guess you had to be there.
You can’t appreciate a feat of organization until you experience the disorganization. But it’s hard to have the patience to wrap your head around a disorganized mess that you don’t care about. Only if the disorganized mess is your responsibility, something that means more to you than a case study, can you wrap your head around it and appreciate improvements. This means that while you can learn algorithmic wizardry through homework assignments, you’re unlikely to learn organization skills unless you work on a large project you care about, most likely because you’re paid to care about it.
Related posts:
Sean Parent gives great talks on C++ that mostly focus on this. An example is http://channel9.msdn.com/Events/GoingNative/2013/Cpp-Seasoning.
But how do you present a clever bit of organization?
– Try to present a problem with high combinatorial complexity, things like we need to support multiple metric systems, drawing different shapes, on different material, and so on, and then present the bridge pattern.
Tell them it is for real, system like that is used in real CAD for both VLSI and cars. Different metrics, different shapes and different materials, for real!
This goes along the Steve McConnell’s “Primary Technical Imperative of Software Engineering”, which is Complexity Management. I’ve been reading a lot about patterns, agile, clean code and the like, and couldn’t find a single suggestion of “best practice” that didn’t involve, at its core, some sort of Complexity Management.
That is to say that I fully agree with the premise of the posted excerpt, and your additional comments on that are very interesting!
This is not even unique to software engineering either! Many of the best ideas are not technical wizardry or invented by the person with the deepest understanding, but by people who manage to create simple concepts and keep building on them in ways that can eventually scale.
There are people with billion dollar businesses that sell curtains and blinds. That’s sheet logistics and organization. Most people could figure out how to make one set of curtains and install them.
It would be great to have tutorials that walk students from a “before state” through an “after state” of various programs. Like an editing exercise for computer programming.
I wonder if the Literate Programming approach would assist in this? At its heart, Literate Programming is about explaining the program which in turn might help additional insights and so simplifications.
It doesn’t have to, of course: any tool can be used in ways it was not intended for.
There is one very common behavior that adds early complexity that can be very difficult to remove: Opening the IDE before the design has stabilized (AKA “doing system design in the code editor”).
This usually happens when high-risk parts of the system design (and/or requirements) are still in flux, but there is pressure to get started implementing the “easy” parts of the system.
There are ways, such as modeling and prototyping, to resist this temptation. Modeling and prototyping are tools that support the requirements and design processes They are for research and problem solving. They are not venues for premature development of the target code.
The one software expense I’ve never been able to adequately quantify to permit including it in the software schedule or budget is the cost of rewriting/refactoring code. Most folks treat it as “free”, as if it were something programmers do on their “free” time. Wrong.
The most direct way to wind up with clean (simple, well-organized) code is to start with a clean set of requirements and a clean design.
Push design complexity and uncertainty into the proper domains for addressing them. Well away from the code editor.
Stalin said, “Quantity has a quality all its own.”
I agree with BobC’s opinion. However, it isn’t in fashion and contradicts the modern agile approach, which nearly always degenerates into cowboy coding. If all our programs were like ‘Hello World’ then we’d all be laughing, but every feature a boss or user wants to add, increases complexity and adds costs. It’s easy to write a use case, but no-one wants to admit the costs of all the features mentioned in passing. In other words it is easier to say rather than do and that is the real problem in software development.
Very interesting indeed.
From my experiences, it takes a system to be developed 3 times over before you have a final robust system. It’s very difficult to refactor a system when you are still figuring out the solution space.
If you rebuilt a system, each time you reduce the problem space. It rebuilt reduces and manages the previous iterations complexity.
So what’s the best way to manage complexity? In my mind rebuild it!
“Keep the good parts, and throw out the bad”
I couldn’t agree with you more.
I am often tasked with making changes to large LOB applications that many people have worked on over several years, and it may take me a day or two to change a few lines of code.
This is because I believe any change should be localised to the requested functionality *only*, without any unintended consequences. As such, I have to spend time understanding how a possible labyrinth of interconnected logic paths work prior to changing anything.
“The most direct way to wind up with clean (simple, well-organized) code is to start with a clean set of requirements and a clean design.”
I disagree…
The problem is, software development is not a linear process. It’s not like you do all the brain work up front, and then you just sit down afterwards just to type it into the computer.
Instead, it’s an iterative process.
The more you can pin down up front, the better of course. Up to a certain extent. At some point it becomes more expensive to do additional planning up front, than to just sit down and implement it.
I say a good approach would be to spend about 20% of the time doing planning up front, and then to spend the remaining 80% of the time doing actual implementation work and design through an iterative process.
I guess many “software architects” would disagree with me. Personally I’m not too fond of the job title “software architect”. It sort of implies that architecture and design is a separate step in the software development process. In my experience it’s not. Instead, to create good software, it’s necessary for architecture, design and implementation to be an interwoven process.
As I always say – you are writing the code for other engineers, NOT computers. A computer language is like any other language. Every file tells a story. If you can’t “write good” and structure the story in a way that’s understandable to other people, then you end up with a mess.
A compiler knows what you mean if it compiles. Humans don’t.
Ideally, your code should be so clear that someone reading it will say “this is simple — I don’t see what the big deal is.”
This takes talent and effort. As Pascal wrote: “I would have written a shorter letter, but I did not have the time.”
A few times, I have been developing something on a platform new to me that had perplexing bugs, until I decided that I could not continue trying to understand it until I organized the code into separate methods. The last time I did that, the act of organizing the code magically fixed the bugs. The first rule is to make small functions/methods with clearly defined purposes, then use them as building blocks. I don’t think it would be hard to give students a few examples of spaghetti code and ask them to fix it this way.
So the more important skill is “caring about what you’re doing” or “Giving a sh*t”…
http://mikemainguy.blogspot.com/2015/04/the-hazard-of-not-taking-things.html
Isn’t “organizational skill” just composed of “design/architecture skill” and “will to improve”?
I wonder if we could use refactoring exercises to help determine organizational tendencies the way we use current interview questions to determine algorithmic aptitude?
Not saying I like the current interview questions as an overall programming aptitude metric, but it might be a start?
The most important skill in software development is not the act of coding. It’s the act of designing such that the coding essentially falls out of the architecture.
Screw refactoring and all that. If you’re refactoring, you did the design wrong to begin with (if you even did any design.)
Software development is not a linear process, it is iterative as ArntS says. The key is to perfect the iterative process so that the whole does not degrade after each iteration, especially since it is possible to either hold or improve after each iteration.
The problem is that the key to perfecting the iterative process is not really a technical one, it is a question of discipline on the part of the organization. If your developers and product owners can tell when there is a problem (and if they can’t then you need new ones), then the discipline to halt new iterations until the current one is perfected is required.
To a lot of organizations and managers that seems like a waste of time and money, but they are incorrect and end up paying dearly after only a short time.
When I write code, while I do care about algorithms, managing complexity adds up in the development process. While I have managed this complexity through well-written comments in the code as well as a separate documentation, it is very time and resource consuming, nevertheless. I find myself gravitating toward code / framework that enhance simplicity these days.
Does anybody know or use any specialized tools that would manage code management and complexity?
Static analysis tools will give you various metrics on your code that will point you in the right direction. There is no ‘golden answer’ to good code, but these tools may point you in the right direction; make you think a bit more about some of your design decisions.
This is an equally good tip for new programmers to establish productive habits early on.
This is where I often find the cult of unit testing to be somewhat myopic. Yes, in a relatively simple application, unit testing may catch a lot of problems. But in a system with a high degree of complexity, and a large number of features whose intersections may be nearly infinite, regression and integration testing is much more important. Examples of such complex systems include a text composition engine, a web browser, an image processor, a compiler, or just about anything that deals with complex resources such as graphics, fonts, or scripts at a low level.
Thanks for such great article.
Both skills are needed. And both can be improved on over time.
Implementing algorithms help you with ample organizational skills.
Out of the 29 comments (at the time of my posting), only Dan Korn mentioned unit testing. The more I read about test driven development, the more I’m convinced it’s the way to go.
For existing projects that did not have tests, it certainly becomes harder, but not impossible. “Work effectively with legacy code” seems to nicely cover this and their only solution is to write tests but slowly breaking apart the dependencies of a complex codebase and testing it bit by bit.
I agree that organisation skills are very important, but I personally find that one of the best ways to learn about organisational skill is by learning algorithmic wizardry. The reason design patterns have sprung up over time is to help improve efficiency, remove coupling and improve the organisation of code. By reading a blog, following a tutorial or listening to a podcast, a developer can add a new design pattern to his arsenal. When the time comes to develop a new complex product, they can take the algorithmic lessons they have learnt and apply them to create more organised, maintainable code.
Take for example the use of the actor model (via the Orleans framework) in producing the game Halo 4. A modern computer game is a large and complex product, and by the choice of such a pattern a lot of the complexity of the code, and potential pitfalls around state and concurrency were mitigated. The fact that the developers on the product knew of the actor model, lead them to a well organised solution.
The lost art of simplicity …
TDD does create pressure to go in the right direction, but won’t make you actually follow the hint, unless you’re a lazy bastard who is willing to go to great lengths in thinking things through just to save manual work downstream.
I don’t think patterns and algorithmic knowledge are the same thing, or push you in the same direction. In fact, I don’t think algorists or pattern doctors are the right people to put in charge of keeping things simple and well organized – people obsessed with code smells and refactoring are. Patterns and algorithmic knowledge are useful tools to keep architecture clean, but tools are useless as long as you don’t have a plan for what you want to build.
@Mark Clifton: the one always true thing about software design is that you’ll never get it right on the first attempt. Even if you get it half right, after some years it will still become no longer right, even if carefully maintained, because the context changes. You _have to_ find ways to fix design iteratively, otherwise software development is hopeless and doomed.