VPN Testing Reveals Poor Privacy and Security Practices, Hyperbolic Claims

We evaluated the privacy and security of 16 virtual private networks. Here's where most of them fell short.

Illustration of a laptop with a secure lock on it thats being corrupted digitally Illustration: Miguel Porlan

Whether you’re working from home or just spending more time online, you may have considered using a virtual private network, or VPN, to boost your privacy and security.

A VPN is a service that routes all of the data sent to and from your computer or phone through the VPN provider’s own servers, or servers it rents. You download some software onto your device, and when you’re logged in, the VPN can offer some protection when you’re using free WiFi at an airport, library, or coffee shop by making it harder for network administrators to see what you do online.

Using a VPN can also prevent your mobile network and internet service provider from seeing and tracking the internet domains you visit. The catch is that the VPN itself has access to all of this data, instead.

It can be tricky to pick a good VPN, which is important because you are entrusting it with your personal data. When CR tested the privacy and security of 16 VPN services running on Windows 10, we identified what we consider to be the three best VPNs for most consumers. But more broadly, we confirmed what many security experts have said for years: The industry’s privacy and security practices often don’t live up to its marketing. 

More Digital Security and Privacy

Our analysis was detailed, and a full report is posted on the CR Digital Lab site. In brief, we looked to see whether security best practices were being used; whether data was being leaked, making it vulnerable to criminals; and what the privacy policies said about how personal data might be used.

We also looked at all the consumer-oriented language on the VPNs’ websites. There’s a lot of hyperbole and unrealistic promises to be found in some VPNs’ marketing—and if you take all of it at face value, you might think you have more protection than you really do.

Signing up for a VPN that’s not doing a great job can make your private data less safe, not more. And not everyone needs a VPN.

“By using a VPN, a user essentially transfers trust from their network provider to the VPN provider. With a misalignment of incentives between users and VPN providers, this can result in bad actors exploiting and preying on users’ trust,” says Roya Ensafi, assistant professor at the University of Michigan and principal investigator of VPNalyzer, an interdisciplinary research project that aims to analyze the VPN ecosystem.

Sweeping Claims

We found that 12 out of 16 of the VPNs we looked at either inaccurately represented their products and technology, or made hyperbolic or overly broad claims about the kinds of protection they provide their users. These claims, combined with a constant stream of news reports about security breaches, can lead people to worry more than necessary about banking online or visiting websites that are already encrypted using HTTPS. They can also give VPN users a false sense of security if they don’t realize that the protections offered are not comprehensive.

“Hyberbolic claims and overpromising by VPNs are not only unethical but also dangerous because it can lead to fostering a false sense of security,” says Reethika Ramesh, a graduate student in Ensafi’s lab and lead researcher at VPNalyzer.

For example, many VPNs have promised complete anonymity or untraceability, or protection from advertisers, governments, and criminals. However, advertisers and governments are both able to track people in many ways not involving their IP addresses, which is what VPNs obfuscate. And your data can be compromised through phishing, malware, and various other methods that VPNs can’t address.

When Ensafi and her colleagues surveyed more than 1,200 VPN users, they found that people placed too much faith in the technology to keep them safe. “Our survey results suggest that even users with high security and privacy expertise express that they feel safe with a VPN, suggesting that VPNs are successful in their marketing efforts,” Ensafi says.

As another example of how VPNs can make overly broad claims, a number of them say they provide “military-grade encryption.” Many security professionals say that this term should be a red flag for consumers, because it doesn’t really mean anything. There is no one standard form of encryption used by the military.

Spokespeople for several VPN companies contacted by Consumer Reports say that broad product descriptions may arise because the companies are trying to explain complicated nuances in just a few words. And, they point out, if you dig deeper on their websites, you can often find additional information that explains that VPNs are just one layer of protection, that points out the VPNs’ limitations, and that tells users to take additional steps to stay safe. Others pointed to additional products offered, some of which are included with a VPN subscription.

However, Consumer Reports found that four VPNs—Mullvad, IVPN, Mozilla VPN, and TunnelBear—do a very good job of describing their products, and giving practical advice on what other services and techniques consumers should use to stay safe.

Difficulty Canceling

We didn’t include evaluations of how easy VPNs were to use in our testing, but one problem stood out. A number of the services made it difficult to turn off auto-renewal or to cancel. This is a problem with services of many kinds, and it can be extremely frustrating.

The VPNs where this was a problem include ExpressVPN, NordVPN, PureVPN, and Surfshark.

To unsubscribe from NordVPN, the consumer first had to click several times and then access an email confirmation to complete the cancellation process. And the email expired in 15 minutes. For ExpressVPN, we needed to select “turn off automatic renewal” a total of three times before it finally worked.

PureVPN and Surfshark were also inconvenient. We couldn’t find any menu that provided a way to unsubscribe from either service. After searching the sites’ support pages to figure out how to cancel the subscription, our tester learned that he would need to send an email and file a support ticket—or cancel through his credit card company or third-party payment processor. 

Privacy

We analyzed privacy policies to determine whether VPNs use their customers’ personal data for purposes beyond just delivering the VPN service. And we looked at how much control users have over their data. This is a blind spot for many consumers. In the University of Michigan survey, about 40 percent of respondents didn’t know that VPNs collect personal data about them that could be used for marketing. We found problems in four of the broad privacy topics that we looked at.

Data Sharing

Some people might worry that a VPN they sign up with might share their personal information with other companies, such as marketers or big data brokers. However, almost every VPN we looked at claimed it would share information with third parties only as reasonably necessary to deliver services—that could mean companies processing your monthly subscription fee, handling automated text messages to your phone, and so on.

Eleven of the VPNs did better, by clearly disclosing the names of all third parties they share information with. That lets both consumers and researchers, at Consumer Reports and elsewhere, learn more about where consumer data might be going. 

One company, IVPN, stands out for stating that no third parties have any access to user data at all, a fact that should appeal to the most privacy-conscious consumers. The company even hosts software it uses to run its operations on its own servers, which is unusual for almost any technology company. That’s important because it reduces the risk of information being compromised by partner organizations. 

Another VPN, Mullvad, generates a random account number for users that does not contain any identifying information about an account and is not connected to a username or email address. And Mullvad does not keep any unnecessary data about its users at all. Even the cookies it uses to keep consumers logged into the service (or remember the language preference or prevent malicious exploits) self-destruct as soon as they close the browser. Only cookies for Stripe, one of Mullvad’s payment processors, remain.

Data Control

We found that every VPN we evaluated could do better when it comes to committing in its terms of service or privacy policy to allowing its users to obtain a copy of all the information that the company holds on them. Many VPN companies say that residents of California or the European Union can access their data, as laws in those jurisdictions (CCPA in California and GDPR in the EU) require them to. But how about everyone else? Only two of the companies—CyberGhost and Mullvad—committed in their documentation to allow users to obtain all of the user data the company holds about them. Even if you live in a place where the law gives you the right to get a copy of your data, you may receive only a portion of it.

Data Retention

Only a quarter of the VPNs that we looked at tell you how long they’ll hang on to all of your data if you’re still a customer—for the others, there’s really no way to know how long personal information will be retained, even if it’s not needed to keep providing you with your VPN service. And only a single VPN, Mullvad, explains how the user data would be handled in the case of a merger, bankruptcy, or acquisition.

Additionally, 14 of the 16 VPNs we analyzed failed to state in their documentation that they will immediately and permanently delete all personal user information when a user closes or deletes their account, and that they’ll do so within 30 days. (Sometimes there’s a valid reason to maintain data after a consumer leaves a service, such as an active legal proceeding.) Even then, the terms specify that old backups containing user data could retain personal information. 

Transparency Reports

It’s a best practice for technology companies to disclose how they share user data with governments, both in the U.S. and around the world, as well as other third parties. This is typically done in a transparency report that includes the number of requests for data the company has received and whether or not it provided the information requested. Transparency reports can help users understand company policies, as well as threats to their privacy and free expression.

However, less than half of the VPNs we analyzed have transparency reports that consumers can find easily. This information is meaningful to many individuals. It can also be important in helping civil society groups understand surveillance laws in various jurisdictions, and also signal which companies are pushing back against improper requests for user data. 

Security

“Insights from our VPNalyzer research and from CR’s testing suggests that the VPN ecosystem lacks accountability due to the absence of good security and privacy industry standards,” Ensafi says.

Consumer Reports’ Digital Lab looked to see whether VPNs had open source code and reproducible builds. This means that the code is posted publicly, so independent security researchers can evaluate it for security flaws, and that researchers can use the source code to recreate the actual working piece of software.

Only six of the 16 VPNs we looked at had reproducible builds, so in many cases, it’s impossible to determine whether the software a user downloads is the same as the code that outside researchers have evaluated.

We also found that only one VPN, Mullvad, uses a signature to authenticate its Windows updates, and just two—IVPN and PIA—use checksums as a data integrity check for them. This means that users of the other VPNs have no way of knowing whether an update is official or has been tampered with. Downloading official software is important because malware peddlers might bundle malicious software with legitimate software in unofficial versions.

Six VPNs we looked at were vulnerable to brute force attacks or account lockouts. Three VPNs allow 30 password attempts without triggering any kind of defense, such as implementing CAPTCHA. The lack of defense increases the possibility of a bad actor getting access to the account. And three VPNs locked out accounts after a number of failed login attempts. This could be abused by malicious actors who could simply enter an incorrect password multiple times for the account they are targeting, ultimately cutting off access to the person who owns the account.

We also found that two VPNs, IPVanish and Kaspersky, were using the deprecated point-to-point tunneling protocol (PPTP), which has serious security flaws and has not been state-of-the-art since the 1990s. 

Oversight

Tech companies frequently discover security problems in their software through internal audits and external, third-party audits. 

For many VPNs, there was no evidence of robust internal procedures for audits, or for preventing unauthorized access by employees—another way that user data can be compromised.

Only four VPNs mentioned internal security audits in their documentation, and only one stated that it publishes summaries of internal security audits. Although eight described protections against unauthorized access, only two had clear and precise descriptions that were up to industry standards.

And half of the VPNs we looked at did not have current, publicly available third-party security audits of their core product. Third-party security audits aren’t a guarantee that a VPN has no security flaws, but they are a sign of trustworthiness, especially if the reports are easily accessible to the public and outside security experts.

Vulnerability Disclosure Programs

A vulnerability disclosure program is a best practice that allows researchers to report security issues they find directly to a company. This makes it more likely that the company can fix the issue before criminals exploit it.   

Though the majority of VPNs have a vulnerability disclosure program for researchers to report security issues, only one company we analyzed has a time frame in its documentation telling people how quickly it will review reports of vulnerabilities, and only three stated without stipulation that they will not pursue legal action against security researchers.

That’s important because security researchers can face legal threats when they find flaws in companies’ software. This harms consumers, because it makes researchers less likely to report security issues that can put VPN users at risk.


Headshot of Electronics freelance writer, Yael Grauer

Yael Grauer

I am an investigative tech reporter covering digital privacy and security. I'm the lead content creator of CR Security Planner, a free, easy-to-use guide to staying safer online. Prior to Consumer Reports, I covered surveillance, online privacy and security, data brokers, dark patterns, clandestine trackers, security vulnerabilities, VPNs, hacking, and digital freedom for Wired, Vice, The Intercept, Slate, Ars Technica, OneZero, Wirecutter, Business Insider, Popular Science, and other publications. Follow me on Twitter (@yaelwrites)