Adam Zerner

Code isn't the only thing that needs to be maintained

When I was working on Premium Poker Tools, it took me a while to launch. I wanted to make absolutely sure that there were no bugs. Even if I launched a free version, I reasoned that bugs would still be catastrophic.

The software tells you things like, "pocket jacks will win 66% of the time against ace five suited". Imagine if the software told you that the number is 56% instead of 66%? You'd lose trust, right? And once you lose trust, you wouldn't want to use the product, right? The whole point of the thing is to tell you what the numbers are. If you can't trust it to give you the right numbers, well, then why use it?

And so, I proceeded to write tests. Lots, and lots, and lots, and lots, of tests. My unit tests took a few minutes to run. Minutes, not seconds.

I also had end-to-end tests. Those took about 30 minutes to run.

To put that in perspective, at my current job where there are maybe 30 engineers and the company has been in business for ~4 years, the tests take a similar amount of time to run.

Part of why I wrote so many tests is also because I was just excited to do so. You see, I learned that testing was this super important thing, but at my first two jobs, I never got to write tests. It was something people acknowledged would be good in theory, but in practice there were always things higher on the priority list to tackle. So, no tests were written in either company.

Then in my side projects, the scope had always been small enough where writing tests seemed sorta silly.

But now I had a sizable project I was working on, where tests were business critical and very appropriate. I wanted to get experience with them. So, I wrote them. Lots, and lots, and lots of them.

"What's the harm?", I asked myself. Ok, maybe some of the tests aren't necessary and I could move a little faster if I avoided writing them. That's ok though, at least I'm getting valuable learning experience and becoming a better developer. Anything for education! Right?

Well, I did learn some valuable lessons, but they weren't about being better at writing tests. They were about being better at not writing tests.

Here's what happened. I reached a point where I would make a change to the code, and all sorts of tests would break. So I'd have to then go through all of those tests and augment them so that they work with my new code change. In the beginning it wasn't so bad, but as the codebase grew and grew, it started to become a bigger and bigger problem. At the end I reached a point where I was afraid of refactoring. That's not a point you want to reach.

That's kinda funny, right? Conventional wisdom is that writing tests gives you the confidence you need to refactor without worrying about breaking things, but my experience was the opposite. I was worried about refactoring because it'd mean I'd have to spend hours and hours going back and augmenting old tests.

I don't want to speak too badly about testing though. I still think of myself as someone who is pro-testing. It's definitely an important thing that has it's place. However, you do have to acknowledge that it comes at a cost. Traditionally people think the upfront cost of writing the tests in the first place. However, I think they often ignore the long term cost of maintaining all of those tests.

Maybe I shouldn't speak for others. I'm not sure what others think. What I can say is that I myself didn't see tests as a thing that needs to be maintained. I knew that code is something that needs to be maintained! I read Bob Martin's book Clean Code and really made an effort to keep my code clean. I knew that when you don't write clean code it comes back to bite you later on.

But tests? That's not the same thing as code. The book isn't called Clean Tests.

When articulated explicitly like this, these thoughts do seem pretty silly. I think the problem is that the questions didn't occur to me. "Should I write clean and maintainable tests?" just wasn't something I consciously thought about. I just assumed that stuff only applied to code, not to tests.

Well, it turns out I was wrong. Tests, like code, need to be maintained.

Recently I had a similar experience, this time with documentation instead of tests. Premium Poker Tools taught me that tests need to be maintained, but I failed to recognize that the same ideas apply to documentation as well.

Throughout the previous 12 months, I started two different jobs. Ramping up and onboarding as a developer is always tough. I often found myself tempted to think, "we need documentation for this". README's, Confluence docs, diagrams − whatever. Just something I could read through to understand what's going on, instead of having to Slack someone or sift through a ton of code. I'm not the only one who is confused, it happens to all new devs getting onboarded. So wouldn't it be more efficient to write it all down once rather than having new devs always asking the same questions?

In theory, yes. But did you notice the mistaken assumption I made?

write it all down once

There it is! That key word "once"! As the codebase evolves, documentation and diagrams grow stale. Just like how test suites also grow stale as the codebase evolves. When things grow stale, you have to update them. And that takes time.

I hope the takeaway from this article isn't "Boo tests!" or "Boo documentation!". I'm still "pro tests" and "pro documentation": I think each are very often underutilized.

The point of this article is just to point out that there are costs associated with both of them. Namely, that they need to be updated and maintained. I think that many people, when they think about costs associated with tests or documentation, they think about those upfront costs of initially writing the specs/docs. What doesn't occur to them, and what didn't occur to me, is the fact that you need to update them when they grow stale, and that over time, this cost will probably end up being larger than the upfront cost.

So then, be smart about when you choose to write tests and when you choose to write docs. Try to get a sense of what the overall, long-term costs will be, and weigh that against the benefits.

If you have any thoughts, I'd love to discuss them over email:

If you'd like to subscribe, you can do so via email or RSS feed.