November 2013 Archives

Hello,

the VPN is back up after experiencing a service outage caused by a DNS amplification attack. The total bandwidth used was about 50Gbit and looked like it was targeting a specific user. Since attacks of that magnitude are becoming more and more frequent we decided to document a few things we picked up along the way in a blog post. If you are not into technical mumbo jumbo feel free to skip the rest of the text.

Debugging

DOS attacks in general are not too uncommon when running a VPN, so we have a number of tools in place that help us to deal with them. The first thing you needs to find out for debugging is which type of attack you are facing. The easiest way is to simply install taps on your uplink infrastructure to be able to see what is going on there. This solution is not an option for us, instead we rely on the ability to see what is happening on the affected servers. It is more selective and less intrusive in respect to our users privacy.

We make sure that the machines are still responsive by having adaptive rate limits configured on the switch ports. Since we know how normal packet rates look like, we simply configure a threshold where the switch starts to drop packets before handing them to the actual server. In reality this works quite well and the packets you get still make sense to find out what is going on. For some servers we have a dedicated administrative interface that allows us to log into the machine when it is under attack. Since switch ports are expensive we do not do that for the VPN servers. There we rely on serial console access which is usually good enough.

A useful optimization we have enabled on our machines is to make sure that the primary service network interface is not tied to the boot CPU with its IRQs. This makes sure that your system stays somewhat usable if you get hit by a DOS that eats 100% of your CPU for IRQ handling.

Looking at the attack pattern we saw that the attacker asked for the A record of cheatsharez.com using a spoofed request pointing to some of our VPN IPs.

$ dig cheatsharez.com
;; Truncated, retrying in TCP mode.

; <<>> DiG 9.4.2-P2 <<>> cheatsharez.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10132
;; flags: qr rd ra; QUERY: 1, ANSWER: 241, AUTHORITY: 2, ADDITIONAL: 2

;; QUESTION SECTION:
;cheatsharez.com.               IN      A

;; ANSWER SECTION:
cheatsharez.com.        21600   IN      A       1.1.1.25
cheatsharez.com.        21600   IN      A       1.1.1.26
cheatsharez.com.        21600   IN      A       1.1.1.27
cheatsharez.com.        21600   IN      A       1.1.1.28
cheatsharez.com.        21600   IN      A       1.1.1.29
cheatsharez.com.        21600   IN      A       1.1.1.30
cheatsharez.com.        21600   IN      A       1.1.1.31
cheatsharez.com.        21600   IN      A       1.1.1.32
cheatsharez.com.        21600   IN      A       1.1.1.33
cheatsharez.com.        21600   IN      A       1.1.1.34
cheatsharez.com.        21600   IN      A       1.1.1.35
cheatsharez.com.        21600   IN      A       1.1.1.36
cheatsharez.com.        21600   IN      A       1.1.1.37
cheatsharez.com.        21600   IN      A       1.1.1.38
cheatsharez.com.        21600   IN      A       1.1.1.39
cheatsharez.com.        21600   IN      A       1.1.1.40
cheatsharez.com.        21600   IN      A       1.1.1.41
cheatsharez.com.        21600   IN      A       1.1.1.42
cheatsharez.com.        21600   IN      A       1.1.1.43
cheatsharez.com.        21600   IN      A       1.1.1.44
cheatsharez.com.        21600   IN      A       1.1.1.45
cheatsharez.com.        21600   IN      A       1.1.1.46
cheatsharez.com.        21600   IN      A       1.1.1.47
cheatsharez.com.        21600   IN      A       1.1.1.48
cheatsharez.com.        21600   IN      A       1.1.1.49
cheatsharez.com.        21600   IN      A       1.1.1.50
cheatsharez.com.        21600   IN      A       1.1.1.51
cheatsharez.com.        21600   IN      A       1.1.1.52
cheatsharez.com.        21600   IN      A       1.1.1.53
cheatsharez.com.        21600   IN      A       1.1.1.54
cheatsharez.com.        21600   IN      A       1.1.1.55
cheatsharez.com.        21600   IN      A       1.1.1.56
cheatsharez.com.        21600   IN      A       1.1.1.57
cheatsharez.com.        21600   IN      A       1.1.1.58
cheatsharez.com.        21600   IN      A       1.1.1.59
cheatsharez.com.        21600   IN      A       1.1.1.60
cheatsharez.com.        21600   IN      A       1.1.1.61
cheatsharez.com.        21600   IN      A       1.1.1.62
cheatsharez.com.        21600   IN      A       1.1.1.63
cheatsharez.com.        21600   IN      A       1.1.1.64
cheatsharez.com.        21600   IN      A       1.1.1.65
cheatsharez.com.        21600   IN      A       1.1.1.66
cheatsharez.com.        21600   IN      A       1.1.1.67
cheatsharez.com.        21600   IN      A       1.1.1.68
cheatsharez.com.        21600   IN      A       1.1.1.69
cheatsharez.com.        21600   IN      A       1.1.1.70
cheatsharez.com.        21600   IN      A       1.1.1.71
cheatsharez.com.        21600   IN      A       1.1.1.72
cheatsharez.com.        21600   IN      A       1.1.1.73
cheatsharez.com.        21600   IN      A       1.1.1.74
cheatsharez.com.        21600   IN      A       1.1.1.75
cheatsharez.com.        21600   IN      A       1.1.1.76
cheatsharez.com.        21600   IN      A       1.1.1.77
cheatsharez.com.        21600   IN      A       1.1.1.78
cheatsharez.com.        21600   IN      A       1.1.1.79
cheatsharez.com.        21600   IN      A       1.1.1.80
cheatsharez.com.        21600   IN      A       1.1.1.81
cheatsharez.com.        21600   IN      A       1.1.1.82
cheatsharez.com.        21600   IN      A       1.1.1.83
cheatsharez.com.        21600   IN      A       1.1.1.84
cheatsharez.com.        21600   IN      A       1.1.1.85
cheatsharez.com.        21600   IN      A       1.1.1.86
cheatsharez.com.        21600   IN      A       1.1.1.87
cheatsharez.com.        21600   IN      A       1.1.1.88
cheatsharez.com.        21600   IN      A       1.1.1.89
cheatsharez.com.        21600   IN      A       1.1.1.90
cheatsharez.com.        21600   IN      A       1.1.1.91
cheatsharez.com.        21600   IN      A       1.1.1.92
cheatsharez.com.        21600   IN      A       1.1.1.93
cheatsharez.com.        21600   IN      A       1.1.1.94
cheatsharez.com.        21600   IN      A       1.1.1.95
cheatsharez.com.        21600   IN      A       1.1.1.96
cheatsharez.com.        21600   IN      A       1.1.1.97
cheatsharez.com.        21600   IN      A       1.1.1.98
cheatsharez.com.        21600   IN      A       1.1.1.99
cheatsharez.com.        21600   IN      A       1.1.1.100
cheatsharez.com.        21600   IN      A       1.1.1.101
cheatsharez.com.        21600   IN      A       1.1.1.102
cheatsharez.com.        21600   IN      A       1.1.1.103
cheatsharez.com.        21600   IN      A       1.1.1.104
cheatsharez.com.        21600   IN      A       1.1.1.105
cheatsharez.com.        21600   IN      A       1.1.1.106
cheatsharez.com.        21600   IN      A       1.1.1.107
cheatsharez.com.        21600   IN      A       1.1.1.108
cheatsharez.com.        21600   IN      A       1.1.1.109
cheatsharez.com.        21600   IN      A       1.1.1.110
cheatsharez.com.        21600   IN      A       1.1.1.111
cheatsharez.com.        21600   IN      A       1.1.1.112
cheatsharez.com.        21600   IN      A       1.1.1.113
cheatsharez.com.        21600   IN      A       1.1.1.114
cheatsharez.com.        21600   IN      A       1.1.1.115
cheatsharez.com.        21600   IN      A       1.1.1.116
cheatsharez.com.        21600   IN      A       1.1.1.117
cheatsharez.com.        21600   IN      A       1.1.1.118
cheatsharez.com.        21600   IN      A       1.1.1.119
cheatsharez.com.        21600   IN      A       1.1.1.120
cheatsharez.com.        21600   IN      A       1.1.1.121
cheatsharez.com.        21600   IN      A       1.1.1.122
cheatsharez.com.        21600   IN      A       1.1.1.123
cheatsharez.com.        21600   IN      A       1.1.1.124
cheatsharez.com.        21600   IN      A       1.1.1.125
cheatsharez.com.        21600   IN      A       1.1.1.126
cheatsharez.com.        21600   IN      A       1.1.1.127
cheatsharez.com.        21600   IN      A       1.1.1.128
cheatsharez.com.        21600   IN      A       1.1.1.129
cheatsharez.com.        21600   IN      A       1.1.1.130
cheatsharez.com.        21600   IN      A       1.1.1.131
cheatsharez.com.        21600   IN      A       1.1.1.132
cheatsharez.com.        21600   IN      A       1.1.1.133
cheatsharez.com.        21600   IN      A       1.1.1.134
cheatsharez.com.        21600   IN      A       1.1.1.135
cheatsharez.com.        21600   IN      A       1.1.1.136
cheatsharez.com.        21600   IN      A       1.1.1.137
cheatsharez.com.        21600   IN      A       1.1.1.138
cheatsharez.com.        21600   IN      A       1.1.1.139
cheatsharez.com.        21600   IN      A       1.1.1.140
cheatsharez.com.        21600   IN      A       1.1.1.141
cheatsharez.com.        21600   IN      A       1.1.1.142
cheatsharez.com.        21600   IN      A       1.1.1.143
cheatsharez.com.        21600   IN      A       1.1.1.144
cheatsharez.com.        21600   IN      A       1.1.1.145
cheatsharez.com.        21600   IN      A       1.1.1.146
cheatsharez.com.        21600   IN      A       1.1.1.147
cheatsharez.com.        21600   IN      A       1.1.1.148
cheatsharez.com.        21600   IN      A       1.1.1.149
cheatsharez.com.        21600   IN      A       1.1.1.150
cheatsharez.com.        21600   IN      A       1.1.1.151
cheatsharez.com.        21600   IN      A       1.1.1.152
cheatsharez.com.        21600   IN      A       1.1.1.153
cheatsharez.com.        21600   IN      A       1.1.1.154
cheatsharez.com.        21600   IN      A       1.1.1.155
cheatsharez.com.        21600   IN      A       1.1.1.156
cheatsharez.com.        21600   IN      A       1.1.1.157
cheatsharez.com.        21600   IN      A       1.1.1.158
cheatsharez.com.        21600   IN      A       1.1.1.159
cheatsharez.com.        21600   IN      A       1.1.1.160
cheatsharez.com.        21600   IN      A       1.1.1.161
cheatsharez.com.        21600   IN      A       1.1.1.162
cheatsharez.com.        21600   IN      A       1.1.1.163
cheatsharez.com.        21600   IN      A       1.1.1.164
cheatsharez.com.        21600   IN      A       1.1.1.165
cheatsharez.com.        21600   IN      A       1.1.1.166
cheatsharez.com.        21600   IN      A       1.1.1.167
cheatsharez.com.        21600   IN      A       1.1.1.168
cheatsharez.com.        21600   IN      A       1.1.1.169
cheatsharez.com.        21600   IN      A       1.1.1.170
cheatsharez.com.        21600   IN      A       1.1.1.171
cheatsharez.com.        21600   IN      A       1.1.1.172
cheatsharez.com.        21600   IN      A       1.1.1.173
cheatsharez.com.        21600   IN      A       1.1.1.174
cheatsharez.com.        21600   IN      A       1.1.1.175
cheatsharez.com.        21600   IN      A       1.1.1.176
cheatsharez.com.        21600   IN      A       1.1.1.177
cheatsharez.com.        21600   IN      A       1.1.1.178
cheatsharez.com.        21600   IN      A       1.1.1.179
cheatsharez.com.        21600   IN      A       1.1.1.180
cheatsharez.com.        21600   IN      A       1.1.1.181
cheatsharez.com.        21600   IN      A       1.1.1.182
cheatsharez.com.        21600   IN      A       1.1.1.183
cheatsharez.com.        21600   IN      A       1.1.1.184
cheatsharez.com.        21600   IN      A       1.1.1.185
cheatsharez.com.        21600   IN      A       1.1.1.186
cheatsharez.com.        21600   IN      A       1.1.1.187
cheatsharez.com.        21600   IN      A       1.1.1.188
cheatsharez.com.        21600   IN      A       1.1.1.189
cheatsharez.com.        21600   IN      A       1.1.1.190
cheatsharez.com.        21600   IN      A       1.1.1.191
cheatsharez.com.        21600   IN      A       1.1.1.192
cheatsharez.com.        21600   IN      A       1.1.1.193
cheatsharez.com.        21600   IN      A       1.1.1.194
cheatsharez.com.        21600   IN      A       1.1.1.195
cheatsharez.com.        21600   IN      A       1.1.1.196
cheatsharez.com.        21600   IN      A       1.1.1.197
cheatsharez.com.        21600   IN      A       1.1.1.198
cheatsharez.com.        21600   IN      A       1.1.1.199
cheatsharez.com.        21600   IN      A       1.1.1.200
cheatsharez.com.        21600   IN      A       1.1.1.201
cheatsharez.com.        21600   IN      A       1.1.1.202
cheatsharez.com.        21600   IN      A       1.1.1.203
cheatsharez.com.        21600   IN      A       1.1.1.204
cheatsharez.com.        21600   IN      A       1.1.1.205
cheatsharez.com.        21600   IN      A       1.1.1.206
cheatsharez.com.        21600   IN      A       1.1.1.207
cheatsharez.com.        21600   IN      A       1.1.1.208
cheatsharez.com.        21600   IN      A       1.1.1.209
cheatsharez.com.        21600   IN      A       1.1.1.210
cheatsharez.com.        21600   IN      A       1.1.1.211
cheatsharez.com.        21600   IN      A       1.1.1.212
cheatsharez.com.        21600   IN      A       1.1.1.213
cheatsharez.com.        21600   IN      A       1.1.1.214
cheatsharez.com.        21600   IN      A       1.1.1.215
cheatsharez.com.        21600   IN      A       1.1.1.216
cheatsharez.com.        21600   IN      A       1.1.1.217
cheatsharez.com.        21600   IN      A       1.1.1.218
cheatsharez.com.        21600   IN      A       1.1.1.219
cheatsharez.com.        21600   IN      A       1.1.1.220
cheatsharez.com.        21600   IN      A       1.1.1.221
cheatsharez.com.        21600   IN      A       1.1.1.222
cheatsharez.com.        21600   IN      A       1.1.1.223
cheatsharez.com.        21600   IN      A       1.1.1.224
cheatsharez.com.        21600   IN      A       1.1.1.225
cheatsharez.com.        21600   IN      A       1.1.1.226
cheatsharez.com.        21600   IN      A       1.1.1.227
cheatsharez.com.        21600   IN      A       1.1.1.228
cheatsharez.com.        21600   IN      A       1.1.1.229
cheatsharez.com.        21600   IN      A       1.1.1.230
cheatsharez.com.        21600   IN      A       1.1.1.231
cheatsharez.com.        21600   IN      A       1.1.1.232
cheatsharez.com.        21600   IN      A       1.1.1.233
cheatsharez.com.        21600   IN      A       1.1.1.234
cheatsharez.com.        21600   IN      A       1.1.1.235
cheatsharez.com.        21600   IN      A       1.1.1.236
cheatsharez.com.        21600   IN      A       1.1.1.237
cheatsharez.com.        21600   IN      A       1.1.1.238
cheatsharez.com.        21600   IN      A       1.1.1.239
cheatsharez.com.        21600   IN      A       1.1.1.240
cheatsharez.com.        21600   IN      A       1.1.1.241
cheatsharez.com.        21600   IN      A       1.1.1.1
cheatsharez.com.        21600   IN      A       1.1.1.2
cheatsharez.com.        21600   IN      A       1.1.1.3
cheatsharez.com.        21600   IN      A       1.1.1.4
cheatsharez.com.        21600   IN      A       1.1.1.5
cheatsharez.com.        21600   IN      A       1.1.1.6
cheatsharez.com.        21600   IN      A       1.1.1.7
cheatsharez.com.        21600   IN      A       1.1.1.8
cheatsharez.com.        21600   IN      A       1.1.1.9
cheatsharez.com.        21600   IN      A       1.1.1.10
cheatsharez.com.        21600   IN      A       1.1.1.11
cheatsharez.com.        21600   IN      A       1.1.1.12
cheatsharez.com.        21600   IN      A       1.1.1.13
cheatsharez.com.        21600   IN      A       1.1.1.14
cheatsharez.com.        21600   IN      A       1.1.1.15
cheatsharez.com.        21600   IN      A       1.1.1.16
cheatsharez.com.        21600   IN      A       1.1.1.17
cheatsharez.com.        21600   IN      A       1.1.1.18
cheatsharez.com.        21600   IN      A       1.1.1.19
cheatsharez.com.        21600   IN      A       1.1.1.20
cheatsharez.com.        21600   IN      A       1.1.1.21
cheatsharez.com.        21600   IN      A       1.1.1.22
cheatsharez.com.        21600   IN      A       1.1.1.23
cheatsharez.com.        21600   IN      A       1.1.1.24

;; AUTHORITY SECTION:
cheatsharez.com.        21600   IN      NS      ns2.cheatsharez.com.
cheatsharez.com.        21600   IN      NS      ns1.cheatsharez.com.

;; ADDITIONAL SECTION:
ns1.cheatsharez.com.    21600   IN      A       89.248.168.94
ns2.cheatsharez.com.    21600   IN      A       89.248.168.94

;; Query time: 23 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Nov 14 03:49:01 2013
;; MSG SIZE  rcvd: 3957

Quite long eh? In an ideal world every DNS server would ask you to upgrade to TCP when you (or someone else) ask for such a long record. Unfortunately this is not the case. Instead some machines connected to the Internet are open resolvers and also try to deliver that message via UDP. Now all an attacker needs to do is to inject spoofed requests (attacker sends an IP packet claiming to be the victim) into the Internet fabric asking for a specific long DNS entry. Luckily for the attacker he can choose from a subset of almost 33 Million open resolvers that will happily answer his spoofed request.

A spoofed request for a DNS record is usually quite small (less than 100 bytes). The reply on the other hand can be quite big, 3957 bytes in our case. If the available bandwidth for the attacker is moderately high enough -- 1GBit is a good ballpark figure here -- you can cause quite some havoc given an average ratio of 1:60. A more detailed explanation of DNS amplification can be found here. To see how those 33 Million hosts spread over the IPv4 Internet have a look at this nice heatmap.

The problem of DNS amplification attacks does not exist since yesterday, reflection attacks themselves have been around for ages. Still we -- we as in the Internet community -- have not been able to fix this particular problematic combination of DNS and spoofing. Looking at the latest statistics from openresolverproject.org we can actually see that the numbers of open resolvers are rising, which is not a good sign.

or-ra-trends.png

What to do in case of an attack

The available options could be generally categorized into:

  1. Disable the network by disconnecting it from the BGP fabric
  2. Move your network to an Anti-DDOS service
  3. Endure the attack and try to mitigate it

Option 1. is the fastest way to deal with the problem at the expense of not being reachable to anybody else anymore. You simply remove the affected network prefix from your uplink ISP lists and the attacker wont be able to route packets to you anymore. Of course this is a rather drastic measure, but it is not too uncommon that your uplink ISP will do that for you anyhow if the attack persists. The issue with DNS amplification in particular is that it can have noticeable effects (as in collateral damage) on the higher level infrastructure.

Given that these kinds of attacks have been around long enough, remain unfixed, and some of the mitigations for attacks of this magnitude are expensive or take a lot of time to implement (more on that later), specialized protection services emerged in the market over the last decade or so. You can opt-in to move your network to somebody who deals with that on a daily basis. cloudflare.com, ddosdeflect.com and voxility are some of the usual suspects when it comes to this kind of service. Those services usually deaggregate the inbound flood using anycast and DPI systems to filter unwanted traffic. Of course this is no option for us -- we certainly will not hand our users traffic to a 3rd party that is located all around the world in all kinds of jurisdictions.

This leaves option number 3. There are a number of things that can be done to reduce the strain of the attack. Some work better than others, some require upfront planning, or money.

Mitigation

Disconnect source name server

From the request above we learned that the domain in question is served by 89.248.168.94 and that the A record has a lifetime of 21600 seconds. Now the nice way is to send an abuse request to the ISP of 89.248.168.94 who is AS29073 Ecatel LTD in that case. The downside is that it might take some time for them to respond. The not so nice way is to suppress the requests from the open resolvers by "disconnecting" the source of the domain from the Internet. After 21600 seconds + whatever caching time is configured on the open resolvers it wont be possible to resolve the DNS record anymore. This will cause the open resolvers to send you failure notices which are not as big as the actual response but still pack a good punch in numbers of small packets. There are a number of downsides with the not so nice way. Collateral damage caused by this action, for example the host in question might be a shared machine. Or you might inadvertently disconnect AS29073 because their uplinks are not very fast and it is just too much traffic entering their network. Furthermore nothing prevents the attacker to switch to a different domain on the fly. Domains are common enough to have plenty of them hosted all over the world. The cat and mouse game of "you annoy us and we annoy you" will be boring pretty fast. Not to mention that the widespread deployment of DNSSEC will deliver DNS records of impressive size for free in the near future, making blocking of the source domain even harder. Keep in mind that the simplicity of the attack allows it to be scaled up easily. It wont help you to incur the wrath of your ISP because he got disconnected by his ISP.

Network upgrades

The next option while expensive is to upgrade the network. While this will not fully mitigate the attack of any size it makes the network more resilient for the smaller ones that happen on a day to day basis. 40G Ethernet equipment is around the corner to be affordable by the lower tier ISPs to bolster their uplinks against such attacks. Of course it might still cost you a dime or two when you decide to accept the traffic at your border.

BGP network prefix management

If the network is multihomed simply choose the provider you know best, and whose infrastructure can handle it to only announce your prefixes there. Make sure you have some beer or other bribes at hand when doing this at night without talking to them first! The network quality will still be shitty (but not fully disconnected) and the whole traffic will enter using the same ISP but it is cheaper on your end for the time being and puts less constraint on the network equipment. Handling 10Gbit incoming junk versus 40Gbit in our case.

If you know you are going to eat DDOS attacks every now and then you can plan ahead and split your network space into smaller chunks. While all the BGP engineers will scream at you because this will eat even more of their precious TCAM memory on their Ciscos, it will allow you to keep parts of the overall network alive and connected if only specific IPs are targeted.

Fixing the DNS server software

Obviously we are not the first people being hit by DNS amplification and others have recognized the problem as well. So far the problem has been proven to be very persistent and remains unfixed. Unlike the other problem classes that we usually deal, like security bugs, to fix this problem we need to change various software components. The changes need to be pushed upstream to the software vendors as well as downstream to the package maintainers of the various Operating System distributions. This is quite a challenging task especially because the changes need to be as easy as possible to be able to push it downstream to all the users running old installations. New features usually only enter the upstream software repositories and it may take years until the majority of machines has them.

We the Internet community should try to push into that direction by "bribing" the responsible people. If it works for Truecrypt and security bugs by handing out bounties shouldn't we try something similar for this problem as well? While the IETF is the correct instance to fix the DNS protocol they are (naturally) slow and adoption takes time.

Maybe the openresolverproject.org project can shed some light on which DNS software versions are used. In any way a structured and coordinated effort to deal with the issue is needed if we do not want to end up with being vulnerable to this particular kind of attack when the next generation of high speed networks come online. Is it a viable goal to patch software so that "If open resolver == true then switch to TCP mode by default by returning truncation needed message"? How about adding some preference to our resolver libs? For example "nameserver tcp:192.168.23.42 [persistent]" to enable adoption of the new scheme? Some resolvers already come with rate limits of some degree but it is not a well defined feature ... DNS caches were designed to be fast and so they are.

Conclusion

As you can see there are not many options to get rid of an ongoing DNS amplification attack. Considering the complexity malware (including the state sponsored one) has reached we can safely assume that there are entities that wield more than 10Gbit injection capabilities. If the attack scales, while maintaining the ratio properties, this could add up to about 600Gbit. Due to its nature it is pretty difficult for any ISP, not wielding the power of the NSA, to find out who is responsible for DNS amplification attacks. A true weapon of internet mass destruction.

Dear users,

the VPN service is currently offline. Unfortunately, a flood of spoofed DNS replies maxed out all of our uplink capacity. Our ISP has disconnected the VPN network range for the time being. We will use the time to do some of the maintenance work that usually requires a downtime.

Once things change we will update you.

The IPredator team