Chinese antivirus vendor Qihoo has been caught out cheating on lab testing by submitting altered versions of its software running AV engines from rival Bitdefender. As a result, the three major testing labs – Virus Bulletin, AV-Comparatives, and AV-Test – have revoked all certifications awarded to Qihoo 360 so far this year and issued a joint statement (PDF) reprimanding the company.
According to the testing labs’ joint press release:
Investigations by the three labs found that all products submitted for testing by Qihoo had one of the product’s four available engines, provided by Bitdefender, enabled by default, while a second, Qihoo’s own QVM engine, was never enabled.
By contrast, as far as can be determined, all versions made generally available to users in Qihoo’s main market regions had the Bitdefender engine disabled and the QVM engine active. According to all test data this would provide a considerably lower level of protection and a higher likelihood of false positives
Considering options are provided within the software to adjust these default settings, some might see the testing labs’ reaction as being a tad pedantic. However, based on the fact that the majority of users tend to leave settings unchanged, the labs have always insisted on using a product’s default settings for testing purposes, and, if these default settings are manipulated, the results are skewed and do not best reflect real-world usage.
“This sort of thing doesn’t really help anyone,” said John Hawes, Chief of Operations at Virus Bulletin. “If the products being tested aren’t those being used in the real world, nobody’s getting any useful information.”
On the face of it, news of this infraction by Qihoo appears to further undermine the value of antivirus testing in general. On the other hand, it is nice to know that the testing labs have jumped all over Qihoo and reacted accordingly. The public submission, rather than some sort of cover up, should go a long way to reassuring antivirus users and vendors alike.
What do you think?