How We Test Antivirus and Security Software

Every antivirus or security suite product promises to protect you from a multitude of security risks and annoyances. But do they actually live up to their promises? When evaluating these products for review, we put their claims to the test in many different ways. Each review reports the results of our tests, as well as hands-on experience with the product. This article will dig deeper, explaining just how these tests work.

Of course, not every test is appropriate for every product. Many antivirus utilities include protection against phishing, but some don’t. Most suites include spam filtering, but some omit this feature, and some antivirus products add it as a bonus. Whatever features a given product offers, we put them to the test.

Testing Real-Time Antivirus

Every full-powered antivirus tool includes an on-demand scanner to seek out and destroy existing malware infestations and a real-time monitor to fend off new attacks. In the past, we’ve actually maintained a collection of malware-infested virtual machines to test each product’s ability to remove existing malware. Advances in malware coding made testing with live malware too dangerous, but we can still exercise each product’s real-time protection.

Each year in early spring, when most security vendors have finished their yearly update cycle, we gather a new collection of malware samples for this test. We start with a feed of the latest malware-hosting URLs, download hundreds of samples, and winnow them down to a manageable number.

We analyze each sample using various hand-coded tools. Some of the samples detect when they’re running in a virtual machine and refrain from malicious activity; we simply don’t use those. We look for a variety of different types, and for samples that make changes to the file system and Registry. With some effort, we pare the collection down to a manageable number, and record exactly what system changes each sample makes.

To test a product’s malware-blocking abilities, we download a folder of samples from cloud storage. Real-time protection in some products kicks in immediately, wiping out known malware. If necessary to trigger real-time protection, we single-click each sample, or copy the collection to a new folder. We take note of how many samples the antivirus eliminates on sight.

Next, we launch each remaining sample and note whether the antivirus detected it. We record the total percentage detected, regardless of when detection happened.

Detection of a malware attack isn’t sufficient; the antivirus must actually prevent the attack. A small in-house program checks the system to determine whether the malware managed to make any Registry changes or install any of its files. In the case of executable files, it also checks whether any of those processes are actually running. And as soon as measurement is complete, we shut down the virtual machine.

If a product prevents installation of all executable traces by a malware sample, it earns 8, 9, or 10 points, depending on how well it prevented cluttering the system with non-executable traces. Detecting malware but failing to prevent installation of executable components gets half-credit, 5 points. Finally, if, despite the antivirus’s attempt at protection, one or more malware processes is actually running, that’s worth a mere 3 points. The average of all these scores becomes the product’s final malware-blocking score.

Testing Malicious URL Blocking

The best time to annihilate malware is before it ever reaches your computer. Many antivirus products integrate with your browsers and steer them away from known malware-hosting URLs. If protection doesn’t kick in at that level, there’s always an opportunity to wipe out the malware payload during or immediately after…