Chinese antivirus vendor Qihoo has been caught out cheating on lab testing by submitting altered versions of its software running AV engines from rival Bitdefender. As a result, the three major testing labs – Virus Bulletin, AV-Comparatives, and AV-Test – have revoked all certifications awarded to Qihoo 360 so far this year and issued a joint statement (PDF) reprimanding the company.
According to the testing labs’ joint press release:
Investigations by the three labs found that all products submitted for testing by Qihoo had one of the product’s four available engines, provided by Bitdefender, enabled by default, while a second, Qihoo’s own QVM engine, was never enabled.
By contrast, as far as can be determined, all versions made generally available to users in Qihoo’s main market regions had the Bitdefender engine disabled and the QVM engine active. According to all test data this would provide a considerably lower level of protection and a higher likelihood of false positives
Considering options are provided within the software to adjust these default settings, some might see the testing labs’ reaction as being a tad pedantic. However, based on the fact that the majority of users tend to leave settings unchanged, the labs have always insisted on using a product’s default settings for testing purposes, and, if these default settings are manipulated, the results are skewed and do not best reflect real-world usage.
“This sort of thing doesn’t really help anyone,” said John Hawes, Chief of Operations at Virus Bulletin. “If the products being tested aren’t those being used in the real world, nobody’s getting any useful information.”
On the face of it, news of this infraction by Qihoo appears to further undermine the value of antivirus testing in general. On the other hand, it is nice to know that the testing labs have jumped all over Qihoo and reacted accordingly. The public submission, rather than some sort of cover up, should go a long way to reassuring antivirus users and vendors alike.
What do you think?
On the surface of AV-C’s allegation/censoring of Qihoo it did seem rather damning for Qihoo. But after reading Qihoo’s public response the matter is not as cut and dry as it first appeared. And I now feel there is culpability on both sides, Qihoo’s & AV-C’s. And here is why.
The test labs claimed Qihoo submitted modified configuation AVs for testing that are not available to the public. Qihoo’s response is that the AVs they submitted were modified configurations, and that they informed the test labs of those changes at the time of submission. Moreover, Qihoo also claim the AVs that were submitted for testing, with the configuration(s) they had, were available to the general public in China. Just not to international users. So if Qihoo is correct, somewhere there was an epic breakdown in communication between the lab(s) and Qihoo. And I think this episode also points to a serious quality issue with the labs themselves.
All this may be true, but would you trust a Chinese company with your data security? Especially one with such a storied past?
https://www.techinasia.com/massive-expose-blasts-qihoo-360-cancer-internet/
Yes, and Qihoo denied all those allegations too, in much the same manner.
Well you do have a point there, Jim. Controversy has surrounded this company, and continues to.
Today I read that Qihoo has decided to withdraw from AV-C.
http://www.hihuadu.com/2015/05/02/360-withdrew-av-c-reviews-traditional-anti-virus-evaluation-criteria-behind-20812.html
I’ve tested a couple flavors of their AVs, but always felt squeamish about keep any on my systems long term.
The simple fact is, you can’t trust the Chinese. They operate under a different set of rules. There is no honor. They will kill babies for money (tainted formula), they will kill pets for money (tainted pet foods), they will do just about anything to get ahead, whether it crosses the boundaries of acceptable, moral or ethical behavior, or not.