Trust but verify - criteria for evaluating VPN providers

Staying anonymous online is like running a marathon through a mine field. We have various government agencies breaking into systems to steal our data, or Internet Service Providers like AT&T that mess with user traffic and inject ads to earn some extra bucks by exploiting their customers for advertising. Besides organizations that make online privacy harder for the general population, technology itself is not easy to deal with either.

There are many pitfalls when using VPNs like:

With all those issues present it is already difficult for the average user to attain even a basic level of privacy or anonymity without spending a lot of time researching how to fix things.

To make matters worse there seems to be a recent trend to splash names on vulnerabilities and then send them rather into the "news on security" circus than doing something about educating users more thoroughly. Portfail comes to mind as the most prominent example. Delivering news in that style sends unsuspecting users into a frenzy because it is hard to filter out what is hype, what are real issues and how do they affect the service(s) used in question. While we agree that identified security issues should be addressed and fixed, there are a number of problems that are rarely spoken about.

Many people are concerned about anonymity but the hard questions are rarely asked or answered for that matter. We have been making this experience with every single VPN provider ranking in the last years that we were asked to participate in. Everybody seems only to be concerned about "the logging question" which cannot be validated at all and which also -- obvious as it is -- is only one question among many others that never get asked at all. To the best of our knowledge there is neither a procedure nor an established entity, that can provide you as a user, with a trustworthy explanation or report based on a real world audit of VPN provider infrastructure. Trust but verify.

So what should users be concerned about? Without any proactive transparency on the VPN provider part, a potential VPN user should look at least for the following things:

Website security

Lets start with the VPN providers website. Its the first line of defense when you need to interact with your VPN provider since you need to make an account, log in, pay for your VPN, etc.

Does the website enforce HTTPS and is it actually properly configured? Use ssllabs.com to check the sites SSL configuration and securityheaders.io to check for HTTP header best practices. Does the website use mixed HTTP/HTTPS content? Imagine the following scenario:

  • HTTPS VPN provider website relies on resources fetched via HTTP
  • VPN provider has servers overseas and the website hosted outside the VPN network
  • You log into the website using the VPN but website access is not routed internally but instead uses the normal internet

In the best case you just face a loss of anonymity by partially exposing communications that reveal things about you. In the worst case you open yourself up to injection attack from the spooks via Quantum Inserts.

Web trackers and affiliate programs

Next on the list is the topic of web trackers. Install NoScript + EFF privacy badger + something like Disconnect. Are there any web-analytics trackers on the website? External resources like JavaScript, social media "buttons", etc? They are all used to track your footprints on the internet.

If the VPN service you are looking at is using ad networks, ask yourself how can you trust someone who is spending money on advertising networks that track you. Are they not supposed to protect you from this exact industry? Many (not all) affiliate programs fall into the same category. How do you pay your affiliates if you do not properly track where your users come from?

Mail system

Next on the checklist is the mail infrastructure. If you have a problem you are likely going to write an email to your provider, so it's a good idea to actually check what they are using.

The rule of thumb here is self hosted == good, externally hosted == bad.

To check the MX record enter the provider domain there. If the result turns out to be Google mail, Hotmail, Yahoo, etc., congrats all your support requests are leaked to a 3rd party. Do not forget that mails contain user identifying information as well. Got something to hide? Read this interesting article and this piece.

Ticket system

The next item on the list when interacting with your VPN provider is to check for a ticket system. While it is nice that they wont "lose" your support requests a ticket systems primary purpose is to NOT forget any customer interaction. Many businesses rely on a ticket system to define Key Performance Indicators (KPI) and consider them essential. If there is a ticket system ask how often old data is deleted.

User data retention

Part of providing a privacy service is that your provider should care about the data you leave in their system. So it's a good idea to check if they have some kind of data retention policy. There are many more places where user identifiable data piles up other than just the VPN itself.

  • Do they delete old user accounts?
  • What about your emails or the ticket system?
  • How many payment records are kept in the system and for how long?
  • Payment logs?
  • Web and mail server logs?
  • DNS server logs?
  • Firewall and IDS logs?

Data a provider does not have cannot be lost. As a small exercise comb through your mail folders or password store and check which of the VPN providers you do not use anymore have deleted your account.

DNS servers

Check if the DNS servers assigned by the VPN are actually located within the VPN itself. Once DNS requests pass network boundaries they are open to manipulation. Quantum insert to the rescue, yet again. The whole issue can be worked around by forcing your system to use DNSCrypt by default.

Jurisdiction

So far all checks have been more or less technical now it's time to look at the organizational aspects of a VPN provider.

The basic question is "What is the primary jurisdiction the VPN provider operates from?". This question is relevant because companies operating from the US or the UK (for example) can be forced to spy on users without having any legal way to disclose that fact to their users. If you are really unlucky the entity operating the VPN service got slapped with a gag order which cannot be violated without risking severe legal consequences.

Some providers try to work around that issue by setting up a warrant canary. A canary is a text file which states that no National Security Letter (NSL) or gag order has been installed by the government. A list of websites having installed warrant canaries can be found at canarywatch.org. While a canary is a good idea, operating a VPN provider in a jurisdiction that does not offer the legal instruments of a gag order and/or forcefully installed network taps is the better way to go.

Server locations

Besides the "We are the most secure VPN in the world" advertising mantra there is the "We are the fastest VPN in the world" motto that gets advertised. Armed with the knowledge from above about jurisdictions you have to ask yourself what counts more. Speed or anonymity? Check where the servers you are connecting to are located network wise by making a whois lookup on the IP addresses. Does the VPN provider actually OWN those machines or are they rented from a 3rd party located in a 3rd (or even 4th) party network?

There is no point in promising you nothing is logged while the network where the VPN server is located in has to be considered hostile. Every infrastructure component a VPN provider hands off to a 3rd party extends the trust relationship you assume to that 3rd party as well. In most cases it lowers the amount of trust you can put into a system rather than increasing it.

In an ideal setting your VPN provider owns all of the hardware AND the network it operates.

What prevents a random 3rd party hoster from actually installing a Management Engine rootkit malware on offered servers for cryptographic key material recovery? Or a network tap for that matter.

The point is that most VPN providers are primarily businesses. The primary objective of most businesses is to drive costs down. In return one of the easiest ways to infiltrate and exploit VPN infrastructure is simply to offer cheap server hosting (do not worry the taxpayer will cover the bills). You might think that this is no big issue, but being on the receiving end there is a never ending stream of hosting offers specifically "tailored for our needs".

Applying the "Trust but verify" principle is really hard considering the environment for 3rd party hosting services.

Below are just two recent examples:

To whom this may concern,

XXX works with many major VPN clients around the globe; so, I wanted
to inquire as to whether IPredator would also have a need for our
extensive US / international server network and vast IP portfolio? 

I am happy to send over additional information at your request.

Many thanks, 
XXX
Hi,

My name is XXX and I represent XXX.com.  I was visiting IPredator.se today and I couldn't tell where you host your VPN nodes.  I wanted to know if you were interested in replacing any of your current US vendors or if you're interested in expanding to a new vendor to diversify where you source your servers.

We have recently begun to specialize in VPN/Proxy company hosting, so we know the nuances of the industry, making the experience pre and post sales efficient and easy for you.  We're happy to announce your IPs or provide our own.  Our servers are covered by a 100% uptime SLA on power and network and a 4 hour hardware replacement SLA.  We have a full featured control panel, SWIP, rDNS, and more.  We make the IP justification process fast and easy.  Our friendly support team is standing by 24/7 365 to assist if the need every arises.

Here are some example quotes, we're happy to tailor these to your needs:

Intel Xeon E3-1230v3
16gb RAM
1TB HDD
20TB bandwidth
1000mbit port
$149/month

Intel Xeon E3-1230v3
16gb RAM
1TB HDD
/24 IP space
20TB bandwidth
1000mbit port
$199/month

We offer services out of our Tier 3 compliant facilities in Dallas, Miami, Chicago, and Los Angeles.  If you're getting better pricing in any of the cities we operate in, we want to see it and beat it!

Thank you for your time and have a wonderful day.

There is another question in that same category: how many of the VPN providers out there are running there services on virtual machines hosted by third parties? Do you think VPN providers can easily offer exit machines in two dozen countries by deploying hardware they own and operate in datacenters they can trust? Honest to good, poke around and just for the fun of it ask some of them.

TLS certificate authentication

Last but not least you should check how your provider authenticates you as a user to their system. Many providers use TLS certificate authentication where a client certificate is issued for your user account. This certificate is then presented to the server infrastructure and used to allow or deny a login to the VPN system. So far so good, but there is a teeny-weeny issue when a TLS certificate is used.

When the TLS protocol suites were designed, anonymity was not seen as important as confidentiality or integrity of the connection. So in reality when you use a certificate to authenticate to OpenVPN for example it will leak the client certificate name and fingerprint in plaintext when negotiating the TLS handshake. This problem has been identified and documented as far back as 2012 but has not been fixed so far (scheduled fix is in TLS 1.3). You might ask whats the big deal here. Assume the following:

  • You create a new VPN account which uses a TLS client certificate
  • You connect from your home IP

Now at that point the information about the association of your client certificate and home IP will have been entered into systems like XKEYSCORE from the NSA, the GCHQ or whatever spook agency which happen to sniff the whole internet just because they can. You go to your best friends house / your spouse / lover / company and connect from there the spooks will know it's you because of the client cert leak. Good bye anonymity ...

Below is an excerpt taken from a raw packet trace that a VPN client sent to its server in a test setup. The name of the client certificate in this test was fbhwaephubsh.vpn.ipredator.se. As you can see this name is also transmitted in plaintext over the wire. This is only a problem in cases where there is a 1:1 relationship between the client certificate and the user. Some providers do not offer any certificate authentication, others use shared client certs, but the general use case for a client certificate is to hand out unique ones.

        0x0110:  0253 4531 1230 1006 0355 0408 1309 4272  .SE1.0...U....Br
        0x0120:  7967 676c 616e 6431 0f30 0d06 0355 0407  yggland1.0...U..
        0x0130:  1306 4f65 6c64 616c 3124 3022 0603 5504  ..Oeldal1$0"..U.
        0x0140:  0a13 1b52 6f79 616c 2053 7765 6469 7368  ...Royal.Swedish
        0x0150:  2042 6565 7220 5371 7561 6472 6f6e 3112  .Beer.Squadron1.
        0x0160:  3010 0603 5504 0b13 0949 6e74 6572 6e65  0...U....Interne
        0x0170:  747a 3127  3025 0603 5504 0313 1e52 6f79  tz1'0%..U....Roy
        0x0180:  616c 2053 7765 6469 7368 2042 6565 7220  al.Swedish.Beer.
        0x0190:  5371 7561 6472 6f6e 2043 4131 2630 2406  Squadron.CA1&0$.
        0x01a0:  092a 8648 86f7 0d01 0901 1617 686f 7374  .*.H........host
        0x01b0:  6d61 7374 6572 4069 7072 6564 6174 6f72  master@ipredator
        0x01c0:  2e73 6530 1e17 0d31 3430 3132 3730 3934  .se0...140127094
        0x01d0:  3234 345a 170d 3234 3031 3235 3039 3432  244Z..2401250942
        0x01e0:  3434 5a30 81a8 310b 3009 0603 5504 0613  44Z0..1.0...U...
        0x01f0:  0253 4531 1230 1006 0355 0408 1309 4272  .SE1.0...U....Br
        0x0200:  7967 676c 616e 6431 0f30 0d06 0355 0407  yggland1.0...U..
        0x0210:  1306 4f65 6c64 616c 3124 3022 0603 5504  ..Oeldal1$0"..U.
        0x0220:  0a13 1b52 6f79 616c 2053 7765 6469 7368  ...Royal.Swedish
        0x0230:  2042 6565 7220 5371 7561 6472 6f6e 3126  .Beer.Squadron1&
        0x0240:  3024 0603 5504 0313 1d66 6168 7762 6570  0$..U....fbhwaep
        0x0250:  6875 6273 682e 7670 6e2e 6970 7265 6461  hubsh.vpn.ipreda
        0x0260:  746f 722e 7365 3126 3024 0609 2a86 4886  tor.se1&0$..*.H.
        0x0270:  f70d 0109 0116 1768 6f73 746d 6173 7465  .......hostmaste
        0x0280:  7240 6970 7265 6461 746f 722e 7365 3082  r@ipredator.se0.

Summary

Never forget: Trust but verify. And if in doubt research.