When unit testing is not enough

June 19, 2008 by pbielicki

I'd like to share my experiences about unit testing (using JUnit) Java servlets outside of the servlet container. Agile world tells us that we should automate as much tests as we can - it would be good if all aspects of the developed system are completely tested. We should test functional as well as non-functional requirements of our systems. But can all tests be automated? What is the REAL value of such tests? The reality brings us problems and pitfalls even experienced developers fall into. I will present such story regarding automated and manual testing of Java servlets.

The problem
Quite recently I had do develop some "proxy" servlets facilitating Ajax request from web browser to our middleware layer. We couldn't directly send Ajax requests because JavaScript security model doesn't allow to request data from other address than it was originally downloaded (similar restriction to Java Applet).

All right then, our servlets were not doing any amazing job (some input and output transformations were needed, however) but I made some bugs even having 100% test code coverage (used Cobertura 1.9).

What about automated tests?
Well some things can be automated, some not. Let's start with unit testing servlets. The simplest (containerless) solution to unit test servlets is to use EasyMock in order to provide mocks for HttpServletRequest, HttpServletResponse, etc. Yes, I know you can use Jakarta Cactus but I did it quicker using easy mock :)

OK - I unit tested my servlets and was very happy that I saw the green light. I started the sevlet container accessed the web page and it seemed to be working. Unfortunately I didn't see that the servlet set some instance variables after the first invocation and these variables couldn't be changed afterwards. The effect was disastrous - servlet was sending the same request all the times not taking into account passed parameters. We could have checked this problem by invoking my doXXX method multiple times with different parameters - we didn't.

Did we test our servlets manually?
Yes, non-automated tests were performed using JMeter. I used JMeter to check how our servlets behave under heavy load. The performance and load results were OK but I noticed in the logs many entries similar to this one:

Checkout date [2008-10-15] is earlier than or equals checkin date [2008-10-11]

Well the log message lies! And the implementation is well-coded (really). What's wrong?
This is how I discovered problem with SimpleDateFormat - not to mention the help from FindBugs that showed me big red light in the affected lines :)

What was wrong?

1. I forgot that using instance variables is not thread-safe (there is only one servlet instance created by the container)
2. I forgot that java.text.SimpleDateFormat has synchronization problems and cannot be used as a static class field
3. I took UTF-8 for granted - unfortunately default encoding was set to the server's default one (this problem was not described above but it appeared during further manual UI testing)

Lesson learned
When you want to test your servlets:

1. You should invoke all methods you test multiple times
2. You should test your servlets using couple of simultaneous threads to see whether there is no shared data clash
3. Be sure you set required content type and content encoding before you flush any response - DON'T take anything for granted
4. You should use FindBugs to check any common Java problems (e.g. SimpleDateFormat) - either using Eclipse/NetBeans plugin or Maven report
5. ALWAYS test your servlets under heavy load - what works for single user can stop working when many users try to use your application

Using load testing tools you can discover quite interesting and unexpected problems - without such test you cannot be sure your application will work under every circumstances.

After implementing all fixes our servlets seem to work very well even under heavy load. We don't have any thread-safety issues and our dates are parsed correctly ;)

Conclusion
To conclude shortly I will say that testing servlets is not easy. You can write unit test, "inject" mock request, response, etc. objects but you still will not be sure how your system behaves inside servlet container under heavy load (you should also try using Jakarta Cactus framework).
Agilists say that you should automate as much tests as you can. I generally agree with it but sometimes it makes more sense to perform some manual testing which simply works. I probably would not discover all the issues I revealed using only automated tests.

If you have other experiences with testing Java servlets, please share them here.

Permalink