June 2013 Archives

The OpenVZ statistics collection feature

Hello *,

there are various means to leak data about a machine to the Internet when using a VPN. Common issues include DNS leaks, or phone home features that certain software (browser toolbars) or devices (mobile phones) add to the challenge of securing your setup.

Every now and then we find something interesting that is worth sharing. Today we take a look at how OpenVZ collects statistics. OpenVZ is a quite popular project aimed at providing soft virtualization features. It integrates nicely with CentOS, Debian and some other distributions.

Lately a new package as part of the OpenVZ installation named vzstats was deployed. Looking at the description of the RPM package distributed for CentOS and co ...

This is an OpenVZ component to gather OpenVZ usage and hardware statistics,
in order to improve the project.

Statistics gathered and reported include the following:
1 Hardware info.
- CPU, disk, memory/swap, PCI devices
2 Software info.
- host distribution, versions of OpenVZ components, kernel version
3 Containers info.
- number of containers existing/running/using ploop/using vswap
- OS templates of containers
For more details, check the scripts in /usr/libexec/vzstats/ directory.

All submissions are anonymous and are not including IP or MAC addresses,
hostnames etc. Global data are available at http://stats.openvz.org

For more details, see:
* http://openvz.org/vzstats
* http://stats.openvz.org

... we learn that quite a few system data is collected and sent anonymously not including our IP or MAC.

What happens under the hood

When you install the package the following things happen:

  • The installer of vzstats connects to stats.openvz.org and gets a UUID to uniquely identify your system
  • curl -F magic_word=very-please http://stats.openvz.org/genuuid.php is the "magic" to do that
  • Once that is know it stores the result in /etc/vz/.vzstats-uuid, the UUID is used for subsequent interactions with stats.openvz.org.
  • The vzstats config is kept in /etc/vz/vzstats.conf (or similar)
  • A collection is run on your system and sent to the collector

So your server talks to some website to generate a unique machine identifier and sends a bunch of data to some statistics host. Automatically if you want it or not. One can disable vzstats (*touch /etc/vz/vzstats-disable*) but you data is at least sent once unless you block the statistics host somehow. Hmm a unique identifier vs. anonymous as the description claims means some kind of tracking takes place. Time to check out what happens under the hood.

The basic mode of operation after installing the vzstats package is to execute the following piece of code on boot:

/etc/init.d/vz start hat:
        # Try to run vzstats to report new kernel
        vzstats >/dev/null 2>&1

Or to execute a cronjob every month. How convenient. Looking at the statistics website [http://stats.openvz.org](http://stats.openvz.org/) we see some nice pictures about how many systems are actively using vzstats to send data. The kernels used, how many containers per host, CPU, memory, disks setup, distribution. Full stop ahead. There is not just the kernel version but also some other not unimportant things that this website knows about and displays in a nicely aggregated format.

Lets see what they are really sending to the statistics collector.

$ file /usr/sbin/vzstats
/usr/sbin/vzstats: POSIX shell script text executable

A shell script ... that makes it easy. Looking at the script we find various functions to deal with the UUID, proxies, some checks and finally the core of whats collected:

for S in $SCRIPTS; do
        NAME=$(basename $S)
        echo "== executing $S ==" >> $OUTDIR/$LOG
        $S > $OUTDIR/$NAME 2>>$OUTDIR/$LOG

$SCRIPTS points to /usr/libexec/vzstats in our case. Inside we find:

$ ls -1

Hum. Quite a bit of data that is collected there. Lets break it down a bit:

cpuinfoCollects /proc/cpuinfo output
dfCollects the file system name and usage
fsThe file system type where OpenVZ resides
lsbreleaseInfos about your distro release level
lspciInformation about all PCI devices of your machine
meminfoThe complete output from /proc/meminfo
osreleaseInfos about your distro release level
ostemplatesThis is a list of the templates you use
top-psA list about the processes that your CPU spends most of its time on
unameThe current kernel patch level
vzlistA list of running containers along with which filesystem they use (no container name included)
vzversion-archA list of versions from the vz tools suite
vzversion-debA list of versions from the vz tools suite
vzversion-gentooA list of versions from the vz tools suite
vzversion-rpmA list of versions from the vz tools suite

Quite the list of information that vzstats sends unauthorized, automatically and periodically to some remote machine. top-ps is of particular interest here:

sh top-ps 
12507 php5
393 mysqld
119 cron
11 init
9 nginx
3273 apache2
530 postgres
397 cron
158 postgres
144 nullmailer-send
444 init
129 cron
106 bash
94 bash
88 bash
1000 apache2
128 cron
73 apache2
73 apache2
16 init
21779 java
832 mysqld
118 cron
20 nginx
11 init
21164 bash
2790 perl
391 mysqld
336 bash
150 cron

The issue

At this point you might ask what the big deal warranting a blog post here is -- after all we are just sending some data about our system to a statistics host that will then generate some nice data.

For all intents and purposes vzstats might be a well intended "feature" to help with the development of OpenVZ. The issue is when one looks at the whole system. To summarize:

  • Sends data unauthorized by the user/sysadmin, automatically and periodically
  • Sends data unencrypted over an unencrypted communications protocol (remember it uses HTTP not HTTPS)
  • Includes a ton of information that gives a nice fingerprint of your machine

Besides all of the basic blunders here there is one more issue that has to be taken into consideration. If you look up the IP address of the statistics host you get This particular machine is located in the USA (you know the country that looks at things through a PRISM).

Lets rephrase the issue a bit: So basically everybody using recent versions of OpenVZ leaks internal data to a machine hosted in the USA where the New Stasi Agency (NSA) -- or similar organizations along the way -- can now collect detailed information about your setup. This means you deliver information which kernel you run, which processes are running on your machine, the kind of CPU you have. Basically everything an attacker needs to know to simplify the process of attacking your server. The data collected by OpenVZ is not just interesting for agencies that sniff our traffic in order to ensure our freedom, but also for a determined attacker who is looking for some easy/juicy targets. The fact that OpenVZs user base to a fair share is made of people using open source to drive alternative, non-mainstream projects, makes it even worse.

Whatever the case is to correlate the data to IP addresses (sniffing or opening the openvz.org box) the amount of information sent to stats.openvz.org makes it easy to target your system. In times where "professional defense contractors" sell exploits to governments we should pay extra attention to not weaken our systems defenses by leaking crucial system information.

Why can't OpenVZ ask the user/sysadmin if he/she/* wants to participate (like Debian and so many other do)? Why can't OpenVZ encrypt the data using GPG/PGP before sending it to the host? Why can't they use HTTPS (at least)?

We tried to talk to the OpenVZ staff but there was not much sensitivity to the issue at hand. Maybe we are just a bunch of paranoid nerds (too much PRISM lately) ... in any case we think if you use OpenVZ you should know about the issue.

Data collection should always be an opt-in feature and not opt-out as it is implemented with vzstats now.

Payment unlocked on beta page

Dear users,

we have unlocked the payment wizard on the beta page. Right now you can pay using:

  • BitPay.com for automatic BTC processing
  • PaySon for european payments via CC
  • Payza and PayPal for international payments

Fully automatic processing of PaySafe cards and a few other payment processors will available in the future. Please test the thing so that we can iron out any remaining bugs and turn off the old website.

The IPredator team