I’m not against Manual Testing, esp. Exploratory Testing. However one needs to consider the following issues with manual testing (checking) as listed below:

• Manual Tests are more expensive and time consuming
• Manual Testing becomes mundane and boring
• Manual Tests are not reusable
• Manual Tests provide limited visibility and have to be repeated by all Stakeholders
• Automated Tests (Checks) can have varying scopes and may require less complex setup and teardown
• Automated Testing ensures repeatability (missing out)
• Automated Testing drives cleaner design
• Automated Tests provide a safety net for refactoring
• Automated Tests are living up-to-date specification document
• Automated Tests dose not clutter your code/console/logs

Time and again, I find developers (including myself) over-relying on our automated tests. esp. unit tests which run fast and  reliably.

In the urge to save time today, we want to automate everything (which is good), but then we become blindfolded to this desire. Only later to realize that we’ve spent a lot more time debugging issues that our automated tests did not catch (leave the embarrassment caused because of this aside.)

I’ve come to believe:

A little bit of manual, sanity testing by the developer before checking in code can save hours of debugging and embarrassment.

Again this is contextual and needs personal judgement based on the nature of the change one makes to the code.

In addition to quickly doing some manual, sanity test on your machine before checking, it can be extremely important to do some exploratory testing as well. However, not always we can test things locally. In those cases, we can test them on a staging environment or on a live server. But its important for you to discover the issue much before your users encounter it.

P.S: Recently we faced Error CS0234: The type or namespace name ‘Specialized’ does not exist in the namespace ‘System.Collections’ issue, which prompted me to write this blog post.

I once worked for a manager (a.k.a Scrum Master) who insisted that we should have at least 85% code coverage on our project. If we did not meet the targeted coverage numbers this sprint, then the whole team would face the consequences during the upcoming performance appraisals. In spite of trying to talk to her, she insisted. I would like to believe that the manager had good intent and was trying to encourage developers to write clean, tested code.

However the ground reality was, my colleagues were not convinced about automated developer testing, they lacked basic knowledge and skill to do effective developer testing and the manager always kept them under high pressure. (Otherwise people would slack, she said.)

Now what do you expect to happened when managers push certain (poorly understood) metric down developers’ throats?

Humans are very good at gaming systems and they do a wonderful job.

• Some developers wrote automated tests with ZERO assert statements
• Others wrote complex, fragile test code which was an absolute maintenance nightmare. Their coverage numbers looked impressive.
• Me and my pair wrote a small test which would simply crawl all the code base. We were able to achieve 99% code coverage. (1% intensionally left, so we don’t look suspicious.)
• And so on…

The manager was very happy and so was the team. This went on for some sprints. One could find our manager showing off to her colleagues about how much the team respected her words and what a command she had on us.

Alas, her happiness lasted only for a few sprints. One fine Sprint demo day, I convinced the team to showcase our dirty little secret to all the stakeholders and management.

The event did shake up the management. It also sent out a clear message.

Code Coverage measures which line of code got executed (intensionally or accidentally) when you run the program or tests against the program. Code coverage tells you nothing about the quality of your tests, hence nothing about the quality of the code. Code coverage along with Cyclomatic Complexity can be used by the team to guide them, but not by management to judge the team.

I regularly practice Test Driven Development. I truly believe in its benefits and also have been able to influence a huge number of develops to practice it.

However, there are situations in which I ask myself questions like:

• Is it important to automate this particular check (test) ?
• Will it be worth the effort/investment or is it YAGNI?
• What is the possibility of this particular scenario breaking (risk)?
• And so on…

Yesterday I was faced with a similar situation where I decided to skip the automated test.

Context: I was working on a website where users can use their credit card while shopping online. Basically the website had a simple text input box to accept the credit card number. It turns out that most browsers (except Safari) caches the input data and stores it in plain text somewhere on the computer.

So we wanted to turn off the auto-complete and caching feature on this specific input field. There is a very easy way to do this. Just need to set autocomplete=”off” in the input tag. For example:

<input type=”TEXT” name=”creditcard” autocomplete=”off” />

Its an easy fix and quite widely used. So that’s great. But how do I write an automated test for this?

If we think a little hard, there are ways to write automated test. But then you ask yourself is it worth it?

This site had lived without this feature for so long, so I did not think it was that crucial to its users. Even if this feature stops works, it won’t bring the site down. (They certainly had a good battery of automated tests which tests the core functionality if the product.) So I choose to skip the automated test. I manually tested it with different browsers made sure it was working as expected.

If this comes back and bites me, I’ll certainly invest in some form of safety net, but for now, time to hack some more code.

What would you choose to do?