Category: testing
In Part 7 of My First Agile Project, I'll be talking about how we went about testing on our project. This part should stand alone if you haven't read the other parts but if you want to catch up, see the series Table of Contents at the end of this post.
Our project was to integrate and configure a new billing system to replace an old custom Oracle forms app. Doing an integration and configuration project presents a lot of challenges to testing, which I'll talk about below. In addition, like all projects, ours had a special set challenges. At first we thought the developers would do most of the real testing on the project with unit tests and verification of configuration changes by our Subject Matter Expert. The original plan included 3 sprints of testing at the end of the project just to make sure we had gotten all the needed functionality in during development. Looking back at this plan now, we all wonder how we could have so stupid. :)
Keep reading for more on how our plans went during development, what happened once we got into testing and why we're still doing testing now - 6 months past our original go-live date. As I said, testing on a configuration / integration project presents special challenges so I hope our adventures will help other teams avoid some of the pitfalls we encountered. If you're doing a similar project or have had different experiences you'd be willing to share with me and others, please post in the comments.
I'd like to share my experiences about unit testing (using JUnit) Java servlets outside of the servlet container. Agile world tells us that we should automate as much tests as we can - it would be good if all aspects of the developed system are completely tested. We should test functional as well as non-functional requirements of our systems. But can all tests be automated? What is the REAL value of such tests? The reality brings us problems and pitfalls even experienced developers fall into. I will present such story regarding automated and manual testing of Java servlets.
The problem
Quite recently I had do develop some "proxy" servlets facilitating Ajax request from web browser to our middleware layer. We couldn't directly send Ajax requests because JavaScript security model doesn't allow to request data from other address than it was originally downloaded (similar restriction to Java Applet).
All right then, our servlets were not doing any amazing job (some input and output transformations were needed, however) but I made some bugs even having 100% test code coverage (used Cobertura 1.9).
At the Agile Open Europe conference I participated in a discussion on the subject of acceptance testing. One of the open questions was how to match "traditional" customers with the agile idea of delivering a potentially shippable product every few weeks.
Different Types of Customers
While some customers are delighted to receive a new release every month, there are others that don't even want to see or know about any of the intermediate releases. Some customers simply insist on working the traditional way. They hire a team of testers for a couple of weeks before the intended release date, and they require the development team to deliver one release candidate, just in time for the test period, and the team should then stop working on the system until the test team has finished testing.
Lately I noticed several discussion threads both offline and online (the most notable online one is in the leandevelopment Yahoo! group) focusing on the value of software testing, both automated and manual. The main discussion point as far as I can see it is in whether the testing is actually a waste that is unfortunately needed to cope with the insufficient practices and should eventually be eliminated.
Knowledge Source
Software development is always not a mass production, but a new product development - software engineers are not manufacturing goods, but rather make a design for a compiler to manufacture something nobody previously built.
I wrote about Acceptance Testing earlier and why it is a very important tool for agile development. In this post, I will talk about how we can make acceptance testing really agile. Actually, it's nothing more special than making Acceptance Testing a part of your Continuous Integration (CI) (an XP Primary Practice). Continuous Integration is the practice of merging, building, and testing code on a continuous basis (almost instantaneous basis).
What is acceptance testing?
Acceptance testing is a black-box testing performed on a software prior to its delivery. It involves running a suite of tests on a completed system (ref: Wikipedia). These test suites are made up of multiple tests or test cases. Each test case consists of a sequence of steps to perform which emulate the use case that is being tested; it also contains input data (if required) as well as the expected output. The result of a test case is either a pass or a fail.
Update: The users of some a bit exotic SDKs reported that they cannot build the example. The reason was the project file SampleTest.mmp. I updated it making the project file a bit redundant. Now example should be compilable on any S60 SDK.
CppUTest is a unit testing framework based on CppUnitLite and designed for the embedded systems usage. Lately it has been ported to Symbian OS used in Nokia S60 smartphones. The framework is xUnit compatible, extremely simple and easy to use. It can print the test results both to console and to a junit-like xml file (later I am going to investigate how well it can be parsed by standard junit output parsers).
Symbian extension. Usage example
Unfortunately, CppUTest package at the moment does not include a full-blown Symbian specific example and lacks support for such Symbian primitives as descriptors (Symbian strings) and leaves (Symbian exceptions).
The file attached to this post contains both. It is a simple and heavily commented example of using CppUTest in Symbian, that includes cpputestsymbianextension.h adding support for descriptors and leaves.
Most of the agile software development methods explicitly or implicitly recommend the use of test-driven-development and generally development of the testable code. The reasons for the high testability demands are no secret. By the very definition of "agility", the agile project has to be ready and even embrace requirement and corresponding architecture changes. Every hardware engineer knows that it is easy to redesign the wired board only if all the components used are reliable and conform to the own specification. It holds true for the software world - it is easy to redesign the system only if its components are well tested.
There is little doubt that the code testability is a good thing, but how can you measure it? Is it possible to have a look at too classes and tell which one is more testable, than another? How to decide that the component is testable enough and doesn't need the further refactoring?
Score-based metric
Jay Flower proposed a score-based metric for testability.
The biggest difference of agile methods from traditional waterfall is the short feedback loop. The whole concept of agility in essence is no more, than "build the most important piece, evaluate, adjust, and repeat". Automated tests in the agile methods serve as a very important tool for shortening the feedback loop.
In the most of traditional processes automated tests are mimicking the manual test procedures. The tests are often written not by the code developers. Tests usually involve testing the functionality of the whole system. Test results come to the original developers quite late and tests don't take into account the latest code changes. Automating functional and system tests is a good thing. It makes testing a little easier, more effective and saves some money for the company. However, it is no more, than automating a small piece of the manual labor.
"To me legacy code is simply code without tests."
Michael C. Feathers “Working effectively with legacy codeâ€
The legacy code is a known software developers’ headache. The legacy code is the difficult to change code that the developers don’t really understand. This code often is inherited from the old developers who left the company, it usually has little to no documentation and little to no testing code. Legacy code can slow down the development speed up to the real competitive problems. When it takes eternity to make a required change, a company can hardly stand a competitor that is able to release life-critical software every month.
RSS feed