Friday, January 4, 2008

Why Vulnerability Counts are a Poor Measure of Security

No of vulnerabilities in 2007: System A-62, System B- 71

Quick, tell me which system is more secure, system A or system B?
If you think that system B less secure, you should look at what those systems really are. System A represents a default installation of Windows XP Home for the year 2007, System B, XP Home with Kaspersky 6.x antivirus and Windows defender. Now tell me which is more secure!
If you mention that these two systems have different software, and a comparison is just not valid, you have got the nub of what I am going to say. Comparing security of different systems by means of vulnerability counts are simply inaccurate, because different vendors fix different things. On one hand, a vendor may just give the OS, another vendor may take responsibility for the OS plus many applications. Taking responsibility for the third party apps will give a raised vulnerability count, but that does not necessarily mean that the second vendor is more insecure. That is because users of the first vendor will have to add third party apps of their own choice and at the end of the installation of all the needed applications, may have an increased vulnerability count. As an example see the table below to see how the vulnerability count for XP Home increases on addition of Microsoft Office 2007, Yahoo! Messenger and Real Player, in addition to Kaspersky 6.x and Windows Defender. Therefore, in many cases, the vulnerability counts of the main vendor usually do not reflect the actual vulnerability footprint of the end user. It should also be noted that many of these applications actually increase the security of the system. Vulnerabilities have been found in the Windows spyware defender and firewall. Does that mean that Vista would have been more secure without them?
Vulnerability counts also assume that a user will use all the software provided by the vendor. This is generally not the case. I use XP and Linux both. On XP, I do not use Internet Explorer for browsing, and use Opera instead. Therefore, I am immune to virtually all the vulnerabilities affecting IE. Opera has fewer discovered security flaws, and almost no reported exploits. However, a vulnerability assessment of my system will show a higher vulnerability count than those who use IE alone, leading to the erroneous conclusion that my system is much more insecure than that of the latter.
Note that plain vulnerability counts do not take into account the responsiveness of the vendor to discovered flaws, nor do they measure the criticality of the flaws. Many vulnerabilities are not dangerous in that they cannot be exploited remotely, or require an improbable series of events to work. Further, when patched, the system becomes safe again. What is important to measure is, therefore, vendor responsiveness. The days-of-risk (the number of days that between the knowledge of a vulnerability and patching) is therefore very important. More important are the days of actual risk , that is the number of days between the posting of an exploit and the release of a patch. Most important is perhaps the user-days-of-risk (the days between release of a patch by the vendor and implementation of the patch by the client). Most exploits in circulation have had patches already released. However, they work simply because a user has not patched the system. Prompt patching, and tools to facilitate them are more important for security.

Chart showing how the vulnerability counts may change when some common applications are added or when just the important holes are counted(the applications included in this graph under "some apps" include Kaspersky 6.x antivirus, Windows defender, MS Office Home and Student 2007, Yahoo!messenger and Real Player) The vulnerabilities have been counted from

Vulnerability counts also do not take account of the fact that many ( I would say that most) of the attacks take place by exploiting the gullibility or lack of knowledge of the user. A system may have almost no critical vulnerabilities, but still be exploited. It also does not mean that a remote attack cannot be carried out on a fully patched system. These need special mitigation measures.
Any vulnerability information is given for a reason. It is meant to help clients stay secure while using their applications. Vulnerability announcements are to promote security. Measuring and comparing (in)security of products is usually subjective and in most cases, goes beyond mere bug counts.


Elie Bursztein said...

Indeed Vulnerabilities count is not a reliable measure. That is why since a few years the research communauty start to use attack surface as security relative metric.