SysAdmin Blog

Hacked very basic support for Alcatel Omniswitch systems into rancid today.

Doesn't do much more than copying the main configuration file verbatim into the repository - none of the error handling and parsing that the other *rancids have, but it's a start.

My perl and expect wizardry is definitely rusty though...

Whoops, no post for over two years... Too much playing EVE Online, not enough system administration, it seems.

Anyway, playing with a new Rancid (cache) 2.3.2a7 installation today gave me the famous "End of run not found" message.

After lots of debugging and mucking around the reason turned out to be depressingly simple: The RADIUS user I had created for Rancid was in a group that didn't have privilege level 15 on the target Cisco device, and wasn't allowed to run commands like "write terminal".

Gnah.

Now I just need to find out what's wrong with the subversion repository.

Damn. Now Versatel (cache) (used to be Tesion) nearly had me with their "no limit" DSL offer. Unfortunately, for me, that's more like "no DSL" - they didn't connect the exchange I'm wired to yet.

That means I'll stay with the Analog-G leased line that I drive with two old US Robotics Courier V.Everything modems (at 33k6) since a few years.



#sh int asy129
Async129 is up, line protocol is up
Hardware is Async Serial
Description: Modem fuer Analog-G zu Alex
Interface is unnumbered. Using address of Ethernet0/1
MTU 542 bytes, BW 38 Kbit, DLY 100000 usec,
reliability 255/255, txload 6/255, rxload 6/255
Encapsulation PPP, loopback not set

...or whatever they are called today.

Perhaps it's a bad case of Not Reading The Fucking Manual, but I just wouldn't expect a so-called package manager (YaST) to downgrade packages which have newer versions installed locally without any safety call back whatsoever.

Just spent a few hours to find out what it managed to destroy on a system I'm thankfully not usually responsible for...

Nothing happens beside hacker meetings this year? Strange...

The atmosphere is decidedly different this time, it seems like less people than in the previous years, which is probably the consequence of several decisions: Higher congress fee, no student rebates, much smaller Hackcenter, overnight stay (sleeping) in the building is discouraged (as is smoking).

Part of the lectures are not really new, some are even just (small) expansions on things that were there last year. Dan Kaminsky go some really interesting results from the things he started last year, though.

WhatTheHack (cache) images are here.

As half of the rest of the world noticed last night, Google had a DNS problem. There's wild speculation about a hack (cache).

As far as I could see from over here, there is no evidence for an attack - Google just seems to have failed in the attempt to diversify their DNS infrastructure.

Currently, all of their nameservers (ns[1-4].google.com) are in their 216.239.32.0/19 network, which may be a bit of a single point of failure if this route is to disappear from the net.

Instead of directly changing the nameservers for google.com in the .com zone, they created the subdomain l.google.com (with nameservers in different networks) and made www.google.com a CNAME to www.l.google.com, probably as a first step before actually updating the 2nd level zones.

The problem was that the nameservers for l.google.com didn't seem to feel authoritative for the subdomain and returned a NXDOMAIN answer for queries about anything in l.google.com. (Although I didn't test it during the outage, it might well be possible that those nameservers also just served the google.com. zone instead.)

Following are admittedly not the most intelligent questions to ask the involved nameservers, but I was just about to go to bed and only wanted to google for something (which didn't work).

www.google.com being redirected to www.l.google.com:

; <<>> DiG 9.2.2 <<>> www.google.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 38832
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0
 
;; QUESTION SECTION:
;www.google.com.                        IN      A

;; ANSWER SECTION:
www.google.com.         686     IN      CNAME   www.l.google.com.
 
;; AUTHORITY SECTION:
l.google.com.           686     IN      SOA     ns1.google.com.l.google.com. 
dns-admin.google.com. 1115309515 900 900 1800 900

the nameservers for l.google.com, with glue from the google.com zone:

; <<>> DiG 9.2.2 <<>> @ns1.google.com b.l.google.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46796
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 4
 
;; QUESTION SECTION:
;b.l.google.com.                        IN      A
 
;; AUTHORITY SECTION:
l.google.com.           86400   IN      NS      a.l.google.com.
l.google.com.           86400   IN      NS      b.l.google.com.
l.google.com.           86400   IN      NS      c.l.google.com.
l.google.com.           86400   IN      NS      e.l.google.com.
 
;; ADDITIONAL SECTION:
a.l.google.com.         86400   IN      A       216.239.53.9
b.l.google.com.         86400   IN      A       64.233.179.9
c.l.google.com.         86400   IN      A       64.233.161.9
e.l.google.com.         86400   IN      A       66.102.11.9

and b.l.google.com not knowing himself:

 
; <<>> DiG 9.2.2 <<>> @64.233.179.9 b.l.google.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 19598
;; flags: qr aa; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
 
;; QUESTION SECTION:
;b.l.google.com.                        IN      A
 
;; AUTHORITY SECTION:
l.google.com.           900     IN      SOA     ns1.google.com.l.google.com. 
dns-admin.google.com. 1115309515 900 900 1800 900

Seems they also missed a dot in the zone's SOA MNAME - so no good QA here :) ...

Google currently (didn't look for history data yet, so don't know if they changed that yesterday, too) seems to have deaggregated the 216.239.32.0/19 network and announce the first couple of /24 as additional routes. This is not considered good practice, but might be a temporary fix to the diversity problem...

This evening I finally got fed up with the diverse web crawlers indexing crap on this server, like the Wiki history pages or the cached pages.

I remembered I had seen a post on the NANOG list about rewriting for crawlers, pointing to this old article on the webmasterworld forums (cache). Starting off from that, I came up with the following rule set for mod_rewrite, which is probably suboptimal, but seems to work in general:

 RewriteCond %{HTTP_USER_AGENT} Ask.Jeeves [OR]
 RewriteCond %{HTTP_USER_AGENT} ^FAST-WebCrawl [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Googlebot [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Gigabot [OR]
 RewriteCond %{HTTP_USER_AGENT} Yahoo\!\ Slurp [OR]
 RewriteCond %{HTTP_USER_AGENT} ^MMCrawler [OR]
 RewriteCond %{HTTP_USER_AGENT} ^msnbot [OR]
 RewriteCond %{HTTP_USER_AGENT} ^ia\_archiver [OR]
 RewriteCond %{HTTP_USER_AGENT} InfoSeek [OR]
 RewriteCond %{HTTP_USER_AGENT} Inktomi [NC,OR]
 RewriteCond %{HTTP_USER_AGENT} ^Scooter [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Teoma [OR]
 RewriteCond %{HTTP_USER_AGENT} VoilaBot
 RewriteCond %{REQUEST_URI} ^/gallery/(do_command|slideshow|poll_results|add_comment|view_photo)\.php [OR]
 RewriteCond %{REQUEST_URI} ^/gallery/.*\?full\=1$ [OR]
 RewriteCond %{REQUEST_URI} ^/tiki/tiki-(pagehistory|view_cache|export_wiki_pages|lastchanges|wiki_rankings|orphan_pages)\.php
 RewriteRule ^/(.*) /dead-end.html? [R=seeother,L]

The REQUEST_URI conditions catch the most annoying requests to the image gallery and to the Wiki part of TikiWiki.

Grrr... after fiddling around for some time with different CDROM drives and such to make my Via EPIA rev.b board boot from CD (which it didn't), I tried setting the ATA CDROM drive to slave instead of master on the secondary bus as last step before destroying the whole thing - and suddenly everything worked.

Should have used PXE netboot from the beginning, which would have saved me an evening or so at least...

Day two was much better, but also some more overcrowded.

• Andreas Bogk, Rüdiger Weis - Trusted Computing, Eine unendliche Geschichte (cache): Andreas and Rüdiger continue their yearly rant about deveopments in the DRM and TCPA arena
• Joanna Rutkowska - implementation and detection of kernel based backdoors and covert channels in Linux kernels (cache): continues my history of missed good talks on 21C3, only heard about 5 minutes. Is said to have been the most technically advanced session on the whole congress
• Thomas Maus - Sicherheitsanalyse von IT-Großprojekten der öffentlichen Hand (cache): actually, this talk was about an implementation for the german "Gesundheitskarte", which turned out to be heavily flawed. Homebrew protocols, no security, useless cryptography, you name it. Unfortunately, the presenter insisted on having questions during the talk, which made it a bit difficult to follow.
• Adam Laurie, Marcel Holtmann, Martin Herfurt - Bluetooth Hacking (cache): the three talked about problems in the implementation of different parts of the bluetooth stack in mobile phones they found, their communcation problems with the industry and the bluetooth SIG, and how to fuck up pretty decent protocols by substandard implementations
• Joi Ito - State of Emergent Democracy (cache): as I'm not into the blogsphere, all this was quite new to me, and I don't think I can summarize the talk into a few words. Wikipedia has an Article on emergent democracy, though
• Donatella Della Ratta - Freedom of expression in new Arab landscape (cache): a quite interesting talk on the developments in arab media since the introduction of Quatar's Al Jazeera satellite news station
• Barry, Mark Seiden (cache): Barry had a stunning demonstration on how to open most of the common locks with a method called "bumping", in seconds. Mark Seiden held a quite generic talk about physical building security, including lots of war stories from his business practice.
• Dan Kaminsky - Black ops of DNS (cache): Dan Kaminsky held a talk about some novel applications for DNS tunneling, like caching streaming data off DNS resolvers, and a user-mode DNS tunneling software (written in perl with Net::DNS) to forward ssh connections to remote hosts (talk dynamic tcp port forwarding through ssh). Quite nice, but his bouncy presentation style is really nerving after an hour or so...
• Felix von Leitner, Frank Rieger - Fnord-Jahresrückblick (cache): almost one of the classics of the recent congresses. Some info on stuff that wasn't really picked up by the mainstream media
• Ray, Stefan 'Sec' Zehl - Hacker Jeopardy (cache): another one of the regular classics, with all of the usual suspects. Went on until 3h in the morning despite some speeding-up in the end.