Anti Virus tests

Abstract

Modern anti-virus tools are a combination of hash databases, pattern matching, behaviour blocking, cloud technology and sandboxes. At least.

Testing them is tricky.

Malware has a short life span. New malware is generated every few minutes (this is called server side polymorphic).

Testing malware is tricky as well.

A testing lab needs access to large heaps of current malware, lots of virtual machines, verification tools to identify if the malware is a dud or dangerous, verified clean samples, …

This is the reason it requires teams, racks of computers, terabytes of malware and cleanstuff (for FP tests) and years of experience.

Computer magazines normally let external experts test the AV software for detection and protection and add their own usability and speed tests on top.

External experts are

• av-test
• av-comparatives
• virus bulletin
  • Which is a AV industry internal organisation

You can get some (not very detailed) raw data from those pages directly. Observe how the different tools develop over time, how they currently achieve, how their detection relates to their False Positives.

Also check out who is not participating in those tests…. this could be a red flag.

Methods

Basically there are two kinds of tests.

Classic scan

A large folder containing historic and current malware is scanned. The percentage of detected malware and maybe the speed of the scan are the output.

The results will be biased because the focus is on historic malware. The dangerous - current - malware is under-represented.

The tested AV products can not use some of their protection tools like URL block lists, behaviour detection, cloud technology, …

This is not used that often anymore.

For that reason this kind of tests is replaced by:

Real-world protection tests

Current malware is used to “infect” machines where a full AV product is running. The antivirus can use all its features to protect the machine.

• URL block list
• Static file scanning
• Cloud scanning
• Behaviour analysis
• Sandbox

The results of this test are much more significant but the test is harder to run and the malware sample set will be smaller.

This kind of tests is the future.

DIY malware tests

Some people do malware tests (by modifying malware, finding some stuff on the internet, …) and do their own testing. Especially patching malware samples to test with “new and unique” samples is a critical thing. Please do not do that. Patching the malware sample could have broken it: it is not running properly and is causing no harm. Not detecting them is the right thing.

Be careful when you see DIY malware tests and always verify with other sources.

Eicar

What you can and should tests is if you antivirus is properly attached to your system.

Basic questions should be:

• Is the proxy intercepting web traffic to scan it ?
• Is the proxy scanning mails when I fetch them with the mail client ?
• Is the AV scanning files when they are stored on hard-disk (called: On-access scanning)
• Is it scanning when I execute a file ?
• Is it scanning when I copy a file to a USB stick ?

Exactly for these kind of tests there is the EICAR test file. Every AV should detect it. And it is easy to detect. It was created for testing exact this kind of situations mentioned above.

EICAR test file

It is just a few bytes long and you can copy & paste it.

Move it around your system and test which entry points are not protected by your antivirus scanner.

Windows Defender

Windows Defender - pre-installed on Windows - is a kind of fallback if no other AV is installed. It improved its detection rate over the last years and already offers a decent baseline. There are still good reasons to install 3rd party scanner. But from an engineering perspective you can expect:

A current Windows with Defender has a decent scanner already installed

There are many drawbacks of having only the Defender as protection: Malware could start to hide from Defender as priority number one. As it is becoming the most common AV product. And this in turn can weaken the real protection Defender is offering. A typical mono culture effect. It is better not to build a mono culture on your systems.