Ryan Bigg

Who? · Books · Blog · Setup · History · Now · Mentoring


13 Feb 2010

Congratulations! You’ve worked so amazingly hard over the last seven-and-a-half months on this project. You met all the client’s demands as they came in. You completed the project on-time and under the $30,000 budget by $2,000. The client is pleased about that. You can tell by the dollar signs in his eyes. The client is also pleased that they’re able to put the website out to the world to grow and conquer it, it already has a couple of users from the beta invites you sent out earlier in the week. You still keep in touch with the client on an almost-daily basis. He’s saying the site’s doing great.


There’s this one user who’s reporting an issue with a feature you worked on 3 months ago. The client has set up his personal email address to receive the bug reports. You receive them when he gets around to sending them. A page is showing “We’re sorry but something went wrong”.

Uh oh.

You log into the website and check for yourself. Sure enough that page is throwing a 500. But why? This feature was completed 3 months ago and worked fine then, didn’t it? You try it on your local machine and you see that perhaps it’s something simple. Perhaps a variable incorrectly spelled in the controller. Or perhaps not. Perhaps it takes you longer than a couple of seconds to identify the issue. Perhaps it takes you an hour, a day.

A week.

So how did this error come to be? You did write tests, didn’t you?

Oh I see.

Because of the impending deadline you were pressured for time and couldn’t be bothered learning any of the testing frameworks. You want to. So many people talk about it in such a positive attitude… it’s just you don’t have the time. When you bring the point up, your boss says “The client’s not paying you to write tests, he’s paying you to write features” Even though your Rails knowledge has increased during the life of this project three to four times the size of what it previously was, you still don’t know shit about testing. Maybe’s the time to start to learn you think. Ah, but The Boss doesn’t want you spending time not producing features. “Tests are a waste of the client’s money” he claims. You sigh. “I’ll do the next project better. I’ll get him to see the light. I’ll really push for testing…”, you say in your head. Ok, so you didn’t write tests.

So here’s the facts:

  • There’s a bug.
  • You worked on it in the past.
  • You must fix it. Now.

So how do you go about debugging this bug to ensure that it doesn’t happen again? Well, you fix it and you know that it won’t happen again because you fixed it, right? You’ll leave a decent comment (actually, you’ll think about leaving a decent comment) after it’s fixed, that way nobody will repeat your mistake. Right?


Alright, so you know it’s sometime in the past. You can view the list of branches and maybe one will trigger your memory, surely you have branches with memorable names?

“Two Strikes” you say? Shame about that. Oh yeah, branching in SVN is a pain in the ass. I agree. If only you could convince your boss to use Git too. Git’s branching is so much nicer. Maybe you’ll get him to “see the light” about that on the next project too. So all the changes are in the trunk, and going back through them is, of course, going to take a lot of time. Best of luck with that.

So let’s assume that you chance across it. You remember the revision and fix the bug. Gone. Dead. Destroyed. Defeated. Sent to the depths of Hell along with one eighteenth of your sanity. That was quite a toughy. You did write a regression test to ensure that it wouldn’t happen again, right?

Oh I see.

You commented on the Basecamp todo item saying that it was fixed. I guess it has a searchable name, right? Just in case it comes up again? It’s called “Something went wrong” or “Page is broken”?

Uh oh.

So you go home, satisfied that you kicked the ass of a bug that had been on your mind for days. You win.


The next day you wake up, grab your coffee/milk/orange juice and check your emails. There’s 3 from Facebook, a couple from a few mailing lists you subscribe to and fourteen from your client. These aren’t fourteen love letters. You see the first one was sent to you 20 minutes after you went to bed. If only you had stayed up later.

You read the first one. It’s a bug report. So is the next one. And the one after that. And the one after that. And guess what? The one after that, too.

All fourteen emails your client sent to you overnight weren’t “Thanks for all your fabulous work on the site” they were the “Why the fuck is this shit broken?” kind of language. He paid good money for it he thinks. You are happy being paid whatever it is you earn. It hasn’t been long that you’ve been doing this anyway and you have to start out somewhere. People make mistakes, right?


You now have bug reports to deal with when you get into work this morning. 8 hours in a day, with each bug taking you 30 minutes to do. Easy, with time to spare. “Pfft. Done by sundown”, you think. You realise that one of the bug reports that got sent in was that bug that you fixed the other day. Another developer working on your project had a conflict with the file and accidentally put back the broken code instead of the working code. Ok. That’s fine. You’ll fix that again. Maybe you never wrote that comment and you insert it now. Yeah, that’ll stop them re-introducing it.

You have a 2hr meeting with another client at the beginning of the day, leaving you with only 6 hours to fix those bugs by sundown. You spend an extra half-hour at lunch. Five-and-a-half-hours. But by lunch you’d already “killed” four of them. You were a bit groggy this morning and you think that lunch will cure that feeling. You’ll conquer the other ten this afternoon.

You don’t.

You finish four, maybe five of them leaving eight, maybe nine bugs fixed. The client is not pleased. You have only done some of the work. ALL the bugs were critical. Your company is small, and you’re the only one with the knowledge of the project, plus, the other developers are busy on their own projects. The client expects… DEMANDS… a 7-day-a-week-24-hour-all-the-freakin’-time response.

Because you’re his bitch.

Alright. So your boss gives him a large apologetic email that you will try harder in the future to meet all of his demands because you don’t want to lose this client. He’s invested $30,000 in this company already and with “Version 2? of the site “just around the corner” he’ll probably invest a lot more.

So you’re back to fixing those remaining bugs. During the night another three have come in. Your bug count is now eight. Eight hours in a day, eight bugs in the list. Done by sundown. You complete five during work hours and two when you get home, after dinner. You’re exhausted. They took you all day to debug. You email the client and say that you’ve only fixed seven of the eight bugs. He emails you back, naturally twenty minutes after you’ve gone to sleep saying that the eighth and final bug “makes the site unusable”. Your boss brings you in in the morning and says that your work isn’t good enough. You baulk at this suggestion. You claim to have fixed thirteen of the fourteen bugs the client emailed you earlier in the week. You feel that you’ve put in some exceptionally hard work this week. Today you’ll have completed the fourteenth one, you’re so confident about that. He naturally asks why there were fourteen bugs in the first place. You don’t have an immediate answer, although you think of mentioning testing. He continues on his rant about that you should be testing the code each time you deploy it. Go through the site and make sure that it’s ok, you know, just click on the links and make sure it’s working. The boss continues on his tirade for an hour. You try to counter his arguments but you view him as some sort of “higher-being” and due to your lack of experience you can’t fault him. He’s the boss man after all. He got there somehow. He’s gotta be right about everything?

Your company has accrued all this technical debt that you didn’t budget for. This carries on for a month longer and now the client has gone over his budget by $2,000. $2,000 he never wanted to spend. Instead of saving $2,000 he’s now spent an extra $2,000 out of his own personal account in order to get this going. He’s talking about leaving your company and moving to another one who could do a better job. This is probably the case. Your boss brings you in and tells you what’s what with this project. The client is leaving. The next day he fires you.

Congratulations, Freebird. You’re out of a job and very, very nearly out of money. A couple of grand is all you have left. Time to begin that hunt again.

So how do you prevent something like this from happening?

You eventually get a new job working with a new company. You sit down with your new client and you do it right. You spec out their application’s stories using something like Pivotal Tracker. You’re taught to write tests religiously for every single story in Cucumber. There isn’t a story without a test. This reduces technical debt in the long run. You won’t have as many bugs like you did when you didn’t test with the previous application. You will have bugs, though, but you won’t be spending those long nights fixing them because you have tests covering it all. The tests will show you where something went wrong because more than likely, a broken part of the site also means a broken test. See a bug that’s not covered by a story or a test? That sometimes happens, but now you have this wonderful framework for really ensuring it doesn’t happen again: Regression testing. Write the test. It breaks. Fix the bug. It passes. If you and your team stick to the Green is Good policy, then you and they will NEVER EVER reintroduce this bug. In theory, anyway. People make mistakes.

So you start off on a simple story such as “User login” and you write a cucumber feature for it with two scenarios. One for logging in. One for logging out. You’ve written all the steps for it but some of them are not implemented, others are just plain broken. That’s because you haven’t written any code for it yet. It may feel backwards at first. This is called Behaviour Driven Development. You’re going to be learning it, and you’re going to do it right from now on. You’re going to be a BDD Zealot. You implement each step one at a time and eventually your feature is all green.

Why do you do this?

So future-you doesn’t come back in 3 months, see that this feature is broken and begin working on a time machine to go back and kick your dumb ass.

Bonus points if you’re using a Continuous Integration server to ensure that this feature and its eventual test brothers stay green. A broken build means a broken application and your singular goal if that build is ever broken is to make it very, very unbroken.


You can now mark that story as “Finished” in Pivotal. Then you can deploy it to a staging server and mark it as “Delivered”. The client then can log in to Pivotal and see that the story is marked as “Delivered” and then log in to the staging server, test that feature and press that lovely holy-grail “Accept” button. You want to be seeing green stories in your Pivotal list. Green is the colour you want to be seeing everywhere.

Green is Good.

So the next feature comes along and involves some rather complex controller logic that returns a JSON hash. You can write this also in Cucumber, or you could write an RSpec Controller test. This is the kind of thing that’s really great for in-team discussions. You’ll suggest Way A of doing it, and maybe the other team people will agree or suggest Way B, and that lovely dynamic of throwing ideas back and forth really produces a better result. Either way, you’ll write the test first and then the code. Behaviour Driven Development. Then the test will pass. Then you can refactor the code if it’s not quite up to scratch and it’ll still pass. Sure it will feel slow at first, but that’s because you’re learning how to do it. Don’t fight it. It will save you a lot of time in the future, where it’s most important. Focus rather on the long term gains rather than the short term.

Eventually after the next seven-and-a-half months you’ll have all these Pivotal Tracker stories all green in the “Completed” section and maybe one or two in the “Backlog” that the client wants done for next release. You’ll run all the specs on your system. They’re all green. You’ll run all the features. They’re all green. You deploy to the staging server for the last time for this release. The client loves it. You deploy it to production and send out the invites. Everyone loves it.


There’s a bug. This time however you’ve installed something like Hoptoad so in your inbox you’ve got the error there, right when it happened. You are the support team for this application, NOT the client. After all, you’re still working for a small company. It contains stacktrace and you can see where it went wrong immediately. You write a test/feature to ensure that this bug would never happen again. It fails because you haven’t fixed the bug. The error is the same one as in the Hoptoad email. You fix the bug.

Congratulations. You’ve successfully regression tested a bug. This bug never happens again for the life of the project.

The client is happy. Your boss asks if he can have a chat with you. You’re doing an awesome job. The other guys were previously on a higher salary than you because you were perceived to be a junior. Not any more. You’ve come an amazingly long way. Now you’re equal. The client wants more work done on his application. He’s flying your boss to his offices next week. They’ve specifically requested you to go along too, as you seem “switched on”. You meet up with the client and discuss the terms of the next iteration, a major one. This deal goes through and your company needs to grow. Massively. The application explodes, in the positive meaning. You run into scaling issues (of course). You bring on another ten guys over the next year just to work on this application. You and a couple of the other “Originals” teach them all you know about your application and the process you’ve helped develop. They then go on to teach the other twenty people you hire in the six months following.

Congratulations. Now you’re doing it right.