Friday, December 19, 2008

Misleading Vulnerability Counts in the Browser Wars

Getting Behind the Raw Numbers

A lot of noise has been made recently about the inclusion of Firefox among the most vulnerable programs on Windows. Certainly, amongst the browsers, it has the dubious distinction of being most hit by vulnerabilities. What is the reason behind this? Is it due to the browser being so buggy that external researchers can find vulnerabilities very easily, or is it due to greater transparency and better internal audit that more bugs are found internally and reported due to its open source nature? To get behind the vulnerability numbers, I therefore decided to find out the externally and internally found vulnerability counts in the browsers for the last year (discovered between 1st January 2008 till now – the 19th of December, 2008). The vulnerabilities were found from the advisories given by the vendors themselves, from the National Vulnerability Database, as well as from the Secunia website.

In terms of Raw vulnerability counts, there is no doubt that Firefox seems to have far too many vulnerabilities. Internet Explorer 7 seems to be the most secure.

However, when broken down into externally and internally found vulnerability counts, it is seen that the apparent difference is entirely due to Firefox patching and giving out details of internal fixes.

It is important that at least the more severe vulnerabilities be found by internal quality control rather than by external sources. The vulnerability severity rating given by the base score of CVSS version 2.0 for those vulnerabilities listed in the National Vulnerability Database (NVD) was used to classify the vulnerabilities into high, medium and low severity bugs according to accepted guidelines. Then the high severity vulnerabilities across the three browsers were compared for both internal or external fixes. It is to be noted that some of the newly released Opera patches do not yet have a CVE number. Therefore, there should be some more vulnerabilities for Opera in the Externally found severe vulnerabilities column. However, the relative order of the browsers will not be changed. The graph below shows the comparison between internally found and externally reported highly severe vulnerabilities for the browsers.


It can be clearly seen that Firefox is not only the best browser as far as documenting internal fixes go, but we are actually assured that they have a well functioning, transparent, internal quality control mechanism in which they actually catch more browser vulnerabilities than its competitors. Now, this again may not be a valid point for comparison since the other browsers are closed source and may actually ship hidden fixes. However, we just cannot have the assurance of internal quality that is given by the Firefox team. Therefore, instead of publicly finding fault with the number of patches in Firefox and comparing the raw, misleading total vulnerability counts, Firefox should actually be congratulated on having such a transparent security process in place, and for patching more documented holes than its competitors.

However, these vulnerability counts should not be used as an indicator of security. Vulnerability counts are a very flawed measure of the security or insecurity of a product. There are better metrics, and rather than the number of vulnerabilities, a composite measure of the number of unpatched days, the efficiency of the patch delivery mechanism, securability of the browser, the potential surface area of attack against the browser and list based protection against malware sites is needed to properly assess the security of the browser. The point behind this article is not that Firefox is the most secure amongst the browsers (that discussion may be left till a later date), but that conclusions based on coarse raw numbers may paint a picture completely opposite the truth. There is a lot to be said about getting behind the numbers.