SysAdmin Blog

Blog Actions

RSS

As half of the rest of the world noticed last night, Google had a DNS problem. There's wild speculation about a hack (cache).

As far as I could see from over here, there is no evidence for an attack - Google just seems to have failed in the attempt to diversify their DNS infrastructure.

Currently, all of their nameservers (ns[1-4].google.com) are in their 216.239.32.0/19 network, which may be a bit of a single point of failure if this route is to disappear from the net.

Instead of directly changing the nameservers for google.com in the .com zone, they created the subdomain l.google.com (with nameservers in different networks) and made www.google.com a CNAME to www.l.google.com, probably as a first step before actually updating the 2nd level zones.

The problem was that the nameservers for l.google.com didn't seem to feel authoritative for the subdomain and returned a NXDOMAIN answer for queries about anything in l.google.com. (Although I didn't test it during the outage, it might well be possible that those nameservers also just served the google.com. zone instead.)

Following are admittedly not the most intelligent questions to ask the involved nameservers, but I was just about to go to bed and only wanted to google for something (which didn't work).

www.google.com being redirected to www.l.google.com:

; <<>> DiG 9.2.2 <<>> www.google.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 38832
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0
 
;; QUESTION SECTION:
;www.google.com.                        IN      A

;; ANSWER SECTION:
www.google.com.         686     IN      CNAME   www.l.google.com.
 
;; AUTHORITY SECTION:
l.google.com.           686     IN      SOA     ns1.google.com.l.google.com. 
dns-admin.google.com. 1115309515 900 900 1800 900

the nameservers for l.google.com, with glue from the google.com zone:

; <<>> DiG 9.2.2 <<>> @ns1.google.com b.l.google.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46796
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 4
 
;; QUESTION SECTION:
;b.l.google.com.                        IN      A
 
;; AUTHORITY SECTION:
l.google.com.           86400   IN      NS      a.l.google.com.
l.google.com.           86400   IN      NS      b.l.google.com.
l.google.com.           86400   IN      NS      c.l.google.com.
l.google.com.           86400   IN      NS      e.l.google.com.
 
;; ADDITIONAL SECTION:
a.l.google.com.         86400   IN      A       216.239.53.9
b.l.google.com.         86400   IN      A       64.233.179.9
c.l.google.com.         86400   IN      A       64.233.161.9
e.l.google.com.         86400   IN      A       66.102.11.9

and b.l.google.com not knowing himself:

 
; <<>> DiG 9.2.2 <<>> @64.233.179.9 b.l.google.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 19598
;; flags: qr aa; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
 
;; QUESTION SECTION:
;b.l.google.com.                        IN      A
 
;; AUTHORITY SECTION:
l.google.com.           900     IN      SOA     ns1.google.com.l.google.com. 
dns-admin.google.com. 1115309515 900 900 1800 900

Seems they also missed a dot in the zone's SOA MNAME - so no good QA here :) ...

Google currently (didn't look for history data yet, so don't know if they changed that yesterday, too) seems to have deaggregated the 216.239.32.0/19 network and announce the first couple of /24 as additional routes. This is not considered good practice, but might be a temporary fix to the diversity problem...