My First Agile Project Part 11: A Tale of Two Dark Clouds

November 10, 2008 by mattgrommes

Picture courtesy of iowa_spirit_walker on flickr

And now, a story about physics. I'll get to programming in a second though, I promise. In 1900 Lord Kelvin (of the Kelvin temperature scale among many other things) spoke at the meeting of Royal Institute in London. There was a consensus at the time that physicists had succeeded in discovering almost everything knowable about the universe. Kelvin was one of the believers in this idea and also one of the most powerful and influential men in science. His speech at the Institute was on 2 'dark clouds' he saw on the horizon, 2 experimental results that didn't fit in with what they knew at the time. Since they thought they knew almost everything, Kelvin was confident that these clouds would be found out and moved past sooner rather than later. What he couldn't have known was that within the next decade, those 2 dark clouds would revolutionize not only physics but would literally lead to the invention of our modern world, including the computer you're reading this article on.

Read on for how this story relates to Agile projects. Although if you've been on a sufficiently large project, you can probably see the connection between things on the horizon that seem small and end up having a huge influence on your project. In this part of My First Agile Project I'll talk about a part of our project that seemed like a 'dark cloud' and ended up being a tornado tearing through our town.

The 2 dark clouds Kelvin was referring to in his lecture were the problems measuring the speed of the Earth around the Sun in what's called the Michelson-Morley Experiment and the difficulty people were having in explaining "black body" radiation. These two experiments led to two of the most important discoveries in science, discoveries you may have heard of even if you're not a science person: Einstein's Theory of Relativity and Quantum Mechanics. This isn't a science article and I'm not a physicist (just a 'science groupie' as Kevin Kelly says) so I won't go into the whys and wherefores of these discoveries. But the fact that 2 unsatisfactory experiments led to theories that we're still investigating today and to many, many inventions should be familiar to any one who has come across backlog items that blew out the schedule or required much more resources than originally estimated. It happens in every field and can break the assumptions and surprise even the smartest of practitioners.

Our Dark Cloud

On our project, our darkest cloud was generating invoices. Our project was to integrate a very customizable off-the-shelf billing system for the insurance company we work for. When we started, we had 3 documents we knew we would have to generate ourselves instead of using the built-in document generation in the billing system because they were too complicated. But even thinking they were complicated, the original plan was for one of our more experienced developers to use the invoice to come up with the system for generating the PDFs and then we would break up the work for the rest of the documents over the course of a few weeks. It turned out, however, that both our assumptions and those of the vendor were conflicting and wrong in different ways. We assumed that being a billing system, it would have built-in mechanisms for generating invoices. The vendor assumed that since every company has different requirements for invoices that each of their customers would naturally figure out their own way of pulling the data they need and generating the invoice.

Once we started into the development of the invoice and discovered that we would have to pull all of our own data from the system we doubled the time we thought we'd need for development of it. A developer from the vendor also started helping. After a while though, we were able to convince the vendor that they should give us a mechanism for pulling some of the data more easily and that would help their next customers as well. So now we could rely on them to make the logic match up between what shows on the screen of the system and what appears on the invoice. This helped with the complexity of the system immensely but by that time we had done so much of the work that converting to their new mechanism actually added time to the project. In the end this solution is a lot more maintainable though since we're not duplicating their logic. Plus, the next customer to come along will owe us some thanks (we wouldn't mind beers and/or money as well :)) for doing this work for them.

Invoices are just plain more complicated than we would have anticipated as well. A billing system has multitudes of different kinds of charges, credits, payments, reversals, writeoffs, and even things I still don't understand like 'negative write-offs' which seems like an oxymoron but thankfully our Product Owner understands them. All of these things need to show up on the invoice, connected to the different policies the customer may have. We also need to make sure the invoice fits on the special perforated paper so the customer can tear off their bottom part and it needs to fit the envelopes and postal service requirements so they don't charge us extra shipping. These are the tear-your-hair-out requirements you never learn until the last minute. Luckily this code ended up being developed by our most experienced programmer so even though there are what seems like a hundred corner cases for what appears where and when on the invoice, the code is easy enough to follow that I've made changes to it without having to spend all day figuring it out.

In the end, this particular dark cloud started out as something we thought would take one developer one sprint and is still not 100% complete after two developers have spent the better part of 9 months on it, including all the various things that have been discovered and added over time. Revamped invoices were part of the reason our company even decided to get a new billing system though so our Product Owner has given it the time it's needed.

The reason I use Lord Kelvin's 'dark cloud' metaphor is to show that even the best and brightest can underestimate what's on or beyond the horizon, often in a big way. Accounting for dark clouds is what Agile does best though. In our case, the invoice is the biggest change our customers will see with the new billing system so it's an important integration point that's worth the effort. We've spent a long time on it but it's definitely worth while and impressive as invoices go. We're going to have less errors on these things than I get on invoices from my cell phone company, that's for sure.

Thanks for reading and I hope this inside look at one big part of our development process was helpful. If you have stories or comments you'd like to share, please do so in the comments below. Thanks and stay tuned for more next week.

My First Agile Project Series
Part 1: Doing 80%
Part 2: Inception & Planning
Part 3: Viral Videos and Bad Jokes in Scrum Demos
Part 4: How to lose credibility and jeopardize your project with lack of management buy-in
Part 5: Our Top 5 Agile Mistakes
Part 6: The First End of Our Project
Part 7: Adventures in Agile Testing
Part 8: 9 Things We Disliked (and Liked) about ScrumWorks
Part 9: Choosing A New Tool - VersionOne
Part 10: 5 Important Issues For Teams

Permalink