Quantity vs. Quality in Security Software Testing

We know how to make better tests of security utilities, but there's probably not enough demand for such tests in the market.

The testing of anti-virus products has always been a tricky business. The better actors in that business are improving their standards, and this presents a challenge in testing.

As I described recently, the most famous testing standards these days are bankrupt in terms of their value to users. The WildList and the VB100 tests which use it exist simply out of ancient tradition and a perceived marketing need not to rock the boat. But these traditions are beginning to break down, and innovative new testing will improve not only what malware is tested against the product, but how it is tested. Some companies, like Trend Micro, have the courage to dump VB100, while others, like Symantec, are "'absolutely' committed to the VB100."

Most anti-malware testing innovation these days, I don't know why, happens in Germany. Andreas Marx of AV-Test not only develops and performs cutting-edge testing, but he has written on the subject extensively. Look for the "AVAR 2007 Conference - Seoul, South Korea" section of AV-Test's Papers page. His paper "Testing of 'Dynamic Detection'" discusses many of the challenges facing testers of anti-malware products in order to meet all that is expected of them, and not just in terms of efficacy.

The old way to look at things focused on quantity. A big part of this emphasis comes from magazines, which fund a lot of the testing. AV-Test specialized in this approach, and could test hundreds of thousands of samples against many products. There is a newer imperative to quantity too: The number of different malware samples in the wild has skyrocketed in recent years. The corresponding number of updates from anti-malware vendors has also skyrocketed.