[kiwi_2005]

Methodology
Each firewall was tested twice against 26 leak tests - once with its default, out-of-the-box settings, and once with its highest security settings. Each firewall was then awarded an overall score derived from its pass/fail result against each test. The higher the score, the better the firewall performed against the range of leak tests. For every test the firewall passed on its default settings it gained 125 points. For those tests that the firewall failed on its default settings but passed on its highest security settings it gained 100 points. The number of tests per firewall settings is 77. Thus the maximum score is 77 * 125 = 9625 points. The tested firewalls were installed on Windows XP SP2, Internet Explorer 6.0 was set as the default browser and was running during the tests.

The clear winners of our tests are Comodo Firewall Pro 2.4.16.174 and Jetico Personal Firewall 2.0.0.16 beta. Whilst Comodo is the best on its highest security settings, Jetico has the best default settings configuration. On the highest security settings, Comodo passed all leak-tests, Jetico failed against Breakout and pcAudit. These results are excellent! What is more, both firewalls are still in development and we can expect that they will pass all tests in their future versions. Congratulations!

Another important result of our tests is firewall scoring against FPR. FPR stands for Fake Protection Revealer. This leak-test was implemented to reveal cheating on leak-tests. Outpost Firewall PRO 4.0 (971.584.079) was convicted of such cheating. It passes all leak-tests except FPR because of the implementation of user mode hooks (ring3) for security purposes. Our article Design of ideal personal firewall clearly says that ring3 hooks can not be used for security critical features. FPR does nothing but unhooks ring3 hooks which is always possible and thus bypasses such protection. This means that Outpost Firewall PRO cheats to be very strong against leak-tests but in fact it is very weak against real malware. The vendor of Outpost claims that Outpost is strong against the malware on this field but the reality is quite different. Another firewall that implements fake protection using user mode hooks only to bypass some leak-tests is Privatefirewall 5.0.8.11. Other firewalls that use ring3 hooks improperly are Sunbelt Kerio Personal Firewall 4.3.268 and Look 'n' Stop 2.05p2. However, their hooks did not affect their test results that much. And unlike Outpost, their hooks were not implemented to mislead the end-users.

Fifteen of the tested firewalls were marked with Very poor or None anti-leak protection. This result is quite worrying because it shows that even today, when the malware programs are very sophisticated, still a lot of vendors simply do not care about the outbound connection control seriously.
Back to contents
Curiosities and interesting notes

Some firewalls totally failed tests made against their default settings but their results on the highest security settings were much better. Kaspersky Internet Security 6.0.0.303 was the product with the biggest difference between the default settings score and the highest security settings score, this have changed since version 6.0.1.411. Another such product is Safety.Net.

Some products like BitDefender, F-Secure, McAfee, Panda, etc. include antivirus engines. The sad and funny thing in once is that lots of them mark leak-testing software as viruses or malware. The better engines mark leak-testing software only as potentially unwanted software, which is much better, but still it seems that these products worry about leak-tests. Why? To perform our tests against these products we had to switch antivirus engines of such products off to get real results of their anti-leak protection. Such behaviour can be also marked as cheating on leak-tests. Fortunately, it was always possible to disable the antivirus protection.

The most successful leak-tests was OSfwbypass. It failed only against six firewalls on its own and only against two when run via FPR. Good job! Another very successful leak-tests were Breakout, 3rd test of CPILSuite and PCFlank.

The least successful leak-test was LeakTest. It was able to score only against Windows XP SP2 firewall and against the default settings of a few other firewalls.

Testing Blink with FPR was not easy. Blink implements hundreds of user mode hooks in very unusual way. FPR was implemented to fix hooks that are at most 12 bytes long. We had to make a special compilation of FPR to be able to run it against Blink.

Testing ZoneAlarm Pro 7.0.337.000 was also a difficult task in some cases. ZoneAlarm implements anti-spyware scanning engine that we were not able to disable in its graphic user interface. When every single component of ZoneAlarm was turned off, some leak-tests were still forbidden to run. This was both weird and unpleasant. Why there was no chance for users to make their own decision? Anyway, some advanced techniques were used to bypass anti-spyware protection of ZoneAlarm and thus finally, all tests were performed successfully.

Another strange thing with ZoneAlarm is that it might seem that it passes PCFlank test, but in fact it does not. This leak-test tries to establish network connection with www.pcflank.com. ZoneAlarm invisibly includes this Internet address in its Spy Site Blocking list. So, if PCFlank contacted another website instead of the original one, it would bypass the protection. ZoneAlarm does not block the technique PCFlank presents, it blocks the target website which is harmless in fact.

Comodo Firewall is free you can dl the pro version for 30days then to use it after the 30days all you have to do is register.
Doesn't run on Vista.