原帖在这里：
http://www.wilderssecurity.com/showthread.php?t=199292
反病毒这个行业，包括这些“知名”测试机构，多几个象Vesselin Bontchev 这样的人，可能某种程度上会好很多。
关于Dr. Vesselin Bontchev：
Dr. Vesselin Bontchev was born in Varna, Bulgaria. He graduated from the Technical University of Sofia in 1985 with an M.Sc. in computer science (systems programming). He worked for the university's Laboratory for Microprocessors and Microcomputers and for the Institute of Industrial Cybernetics and Robotics at the Bulgarian Academy of Sciences, building expert systems. In 1988, he became interested in computer viruses and began producing freeware anti-virus programs. Two years later he became the Director of the Laboratory of Computer Virology at the Bulgarian Academy of Sciences. From 1991 to 1995 he worked as a research associate at the Virus Test Center, University of Hamburg, Germany, where he wrote his Ph.D. thesis on computer viruses.
Since 1990, Dr. Bontchev has been the Bulgarian representative in IFIP's TC-11 ("Computers & Security"). He is also a founding member of CARO (the Computer Anti-virus Researchers' Organization) and a founding member of VSI (the Virus Security Institute).
Dr. Bontchev currently works for FRISK Software International in Reykjavik, Iceland, where he is involved in the development of the anti-virus package F-PROT and is specialized in macro virus research. Outside the field of computer anti-virus research, his interests include cryptography and number theory - he has been a member of the International PGP development team and is participating in the Great Internet Prime Search project.
来源：http://www.people.frisk-software.com/~bontchev/
下面是bontchev的部分帖子内容：
'Perhaps I'm also a bit biased, given that I was the one who personally had to deal with at least part of the crap you were sending to us as "undetected malware".’
'I think I already addressed this issue. Yes, a small fee for a testing company with no other revenue is not unreasonable. Your fee is neither small nor reasonable, though. Hell, you ask for a single (incompetent, crappy) test about as much as I make in a year! And you don't have even a minuscule fraction of the knowledge, expertise and aptitude of what takes to be a good AV tester, let alone an AV researcher.'
'Of course. Nobody is perfect. However, I've had to deal with the "quality" of your tests (or, more exactly, the lack thereof) for several years already and I am convinced now that you don't just "make a few excusable mistakes". Over all these years, their quality hasn't improved an iota. The only thing that has changed is the number of samples used.
Now, I've had the unfortune to study Marxism at school, and there there is the principle that "increased quantities lead to a change in quality" or something like that (not sure how exactly it translates in English). In reality, however, a pile of crap is still crap - even if it is a very big pile.
{snip - Blue}By now I am convinced that you'll never be able to reverse-engineer a piece of malware and understand how it works (or why it doesn't) and that you'll never grasp the fundamental principles on which the various anti-virus products are based.
All you can do is gather a huge pile of files from various dubious sources, run a bunch of scanners on them, process the results in tables (which might even be wrong - I can't tell, because I don't have access to the raw data), and call it a "test".{Snip - Blue}’
'It's difficult to produce exact numbers because this has been going on for years and I didn't keep track of the exact numbers every time. Furthermore, since at FRISK I handle only macro- and mobile malware related issues, I got to see only that part of the "missed" stuff. From what I've heard from the guys at the Virus Lab who had to process the rest, it was of pretty much the same "quality" - but, you realize, this is just hearsay.
Speaking of the stuff I've seen, I can reliably say that at least some 90% of it had no place in a virus test set to begin with. (I mist emphasize - I'm not talking about his entire test set - which I haven't seen as a whole - but only about the stuff that was sent to us as "samples missed by your scanner".) Just because our scanner happened to detect the rest doesn't necessarily mean that they were proper virus samples - but I can't tell for sure without examining them first.
Again, speaking of the stuff I've seen, there were all the typical mistakes made by incompetent virus collectors. Non-functional programs, corrupted executables, files with wrong extension extension (e.g., SIS files with EXE or APP extension; forget which - this is as wrong as renaming a ZIP archive to EXE), partial disinfections, non-working stuff, etc., etc., etc.
I still have somewhere in my backlog a bunch of files labeled "macro stuff from Clementi" which are of such a low quality that they are not only not viruses - they don't deserve even to be classified as "intended". It would be a very rare event (like one in a thousand) that I'd find a genuine new macro virus among the non-working crap. Sadly, this meant that I was forced to analyze it all, looking for these "pearls" - which, as I assume you realize - didn't amuse me at all.
And this has been going on for years, and years. I never saw any improvement. Sure, when we told him "this is crap and shouldn't be in a test set", he removed it without objection. But this is wrong, too. He removed it because he just believed our word. We're a biased side in such things - the proper thing to do is to verify our claims by analyzing the files and confirming what we were saying. Anyway, the crap was removed - only to be replaced with more crap at the next test. As it is, he just shifted to the anti-virus companies the task of sorting out his test set. :-(
A competent tester must always be able to explain why a particular sample is in his test set. "Scanner X detects it", "It was in Company Y's monthly virus collection" and "It was submitted through the FooBar on-line scanning service" are not good explanations. Most empathically not. A competent tester must be able to analyze the sample and say "It is there because this code here performs self-replication and that code there contains the polymorphic engine and my test criteria state that my test set consists of polymorphic viruses".
Andreas, if I pick a random sample from your test set, are you able to analyze it and explain to me exactly what of its properties have validated you placing it there? Because, if you cannot, you're not qualified to test anti-virus programs (or at least not qualified to construct test sets for such tests). And everything that I've seen from the samples coming from you tell me that you're not able or not willing or do not have the time to analyze them. We aren't talking about honest mistakes here. We're talking about mindless pilling of unanalyzed crap.'
......
无法保证样本质量的测试，没有太大意义。 |
楼主: 袋鼠吱吱
发表于 2008-2-15 19:08:41
没有想法
