SysAdmin Blog
SysAdmin Blog
At work, we have a real old (graphite) Apple Airport Base Station (cache), which needed to be configured as DSL router for some wireless clients.
Fine, no problem, fire up Airport Admin on the iBook, configure all the DSL logins and set the thing up for NAT (as I kept the previous ESSID and WEP settings, I had no problem connecting from the wireless side and setting the thing up).
I do all that, see from the Airport status symbol that DSL is connected, but I can't reach anything through the access point (except the base station itself on the internal network).
After much searching around (even dumping PPPoE traffic on the DSL side with ethereal), I still can't find the problem. Connecting with another client works fine, no problems getting to Internet destinations.
Finally, I notice that my iBook's MAC address is not in the access list on the Base Station, and the thing somehow invalidates packets passing through to the PPPoE side. Still, I could connect to and configure the Base Station itself without problems.
Great security, no wonder wardriving was so much fun for quite a time.
Fine, no problem, fire up Airport Admin on the iBook, configure all the DSL logins and set the thing up for NAT (as I kept the previous ESSID and WEP settings, I had no problem connecting from the wireless side and setting the thing up).
I do all that, see from the Airport status symbol that DSL is connected, but I can't reach anything through the access point (except the base station itself on the internal network).
After much searching around (even dumping PPPoE traffic on the DSL side with ethereal), I still can't find the problem. Connecting with another client works fine, no problems getting to Internet destinations.
Finally, I notice that my iBook's MAC address is not in the access list on the Base Station, and the thing somehow invalidates packets passing through to the PPPoE side. Still, I could connect to and configure the Base Station itself without problems.
Great security, no wonder wardriving was so much fun for quite a time.
If you're using a Raptor (Symantec Enterprise) Firewall, it's obviously a bad idea to configure redirects that have a client IP as source, as the firewall will try to do proxy arp for addresses that have redirects (wanted to redirect transparent connections to an arbitrary port on a server instead of the original destination port). Together with client transparency NAT rules, different things will fall over, but not immediately:
Such a combination of redirect and NAT rule can be configured just fine, but after a reboot, an error like the following may pop up:
kernel: 457 ProxyArp Error: A host with address 172.16.3.40 (MAC 00:00:de:ad:be:ef) is active on interface 172.16.3.254 — this address is in use for proxy arp/NAT
...with the side effect that rechability of hosts on the corresponding interface is quite flaky.
So: Don't do it :)
Such a combination of redirect and NAT rule can be configured just fine, but after a reboot, an error like the following may pop up:
kernel: 457 ProxyArp Error: A host with address 172.16.3.40 (MAC 00:00:de:ad:be:ef) is active on interface 172.16.3.254 — this address is in use for proxy arp/NAT
...with the side effect that rechability of hosts on the corresponding interface is quite flaky.
So: Don't do it :)
Last night I noticed our customer backup MX's mailqueue was quickly filling with mails to a certain domain, although the destination mailserver was rechable just fine.
After some searching I found out that there was a (spam) mail to some 100 recipient addresses in the queue, and the destination domain's combination of Firewall I and Interscan Viruswall fucked up on their recipient limitation: After 50 recipients, their server returned 400 class errors on all following commands, even after an RSET, so that our systm couldn't deliver this specific mail and also no other that came later in the queue. Great.
Fortunately, sendmail knows a SMTP_MAILER_MAXRCPTS define, which will break up delivery of mails with more than the maximum of recipients into several sessions (lowered confMAX_RCPTS_PER_MESSAGE too on that occasion - was previously unlimited because of the historical reason that some customers used the MX as outgoing mail server for some mail clients that couldn't limit the number of recipients in a session for CC orgies).
After some searching I found out that there was a (spam) mail to some 100 recipient addresses in the queue, and the destination domain's combination of Firewall I and Interscan Viruswall fucked up on their recipient limitation: After 50 recipients, their server returned 400 class errors on all following commands, even after an RSET, so that our systm couldn't deliver this specific mail and also no other that came later in the queue. Great.
Fortunately, sendmail knows a SMTP_MAILER_MAXRCPTS define, which will break up delivery of mails with more than the maximum of recipients into several sessions (lowered confMAX_RCPTS_PER_MESSAGE too on that occasion - was previously unlimited because of the historical reason that some customers used the MX as outgoing mail server for some mail clients that couldn't limit the number of recipients in a session for CC orgies).
(perhaps I should create an own blog for that topic)
Last evening, I noticed that my DSL router still was running OpenBSD 3.3 (for some unknown reason, I always thought it had 3.4), when I was preparing for the upgrade to 3.5...
Ok, between 3.3 and 3.4 was the big i386 a.out to ELF binary format change, so I thought I'll just reinstall the whole thing with 3.4 and then do the final upgrade... Fine, backed up some config files to the data partition (which I didn't reformat), ran the installer from a disk, copied back a few of the config files , added my changes to rc.local, and then tried to use the DSL connection.
Everything was fine from the DSL router itself (shure, doesn't need NAT), and from another box on the local net. Just the iBook, wich has an extra nat rule with "static-port" flag to make iChat AV work, couldn't get any connection to outside machines. So, I thought, great, the special NAT rule doesn't work for some reason in OBSD 3.4, but why...? pfctl showed no errors when validating and installing the ruleset.
After about an hour of searching, tcpdumping, pfctl'ing, and swearing, I came to the following conclusions:
...and I had forgotten to enable IP forwarding via the appropriate sysctl and the pf rules didn't have anything to do with the problem
Last evening, I noticed that my DSL router still was running OpenBSD 3.3 (for some unknown reason, I always thought it had 3.4), when I was preparing for the upgrade to 3.5...
Ok, between 3.3 and 3.4 was the big i386 a.out to ELF binary format change, so I thought I'll just reinstall the whole thing with 3.4 and then do the final upgrade... Fine, backed up some config files to the data partition (which I didn't reformat), ran the installer from a disk, copied back a few of the config files , added my changes to rc.local, and then tried to use the DSL connection.
Everything was fine from the DSL router itself (shure, doesn't need NAT), and from another box on the local net. Just the iBook, wich has an extra nat rule with "static-port" flag to make iChat AV work, couldn't get any connection to outside machines. So, I thought, great, the special NAT rule doesn't work for some reason in OBSD 3.4, but why...? pfctl showed no errors when validating and installing the ruleset.
After about an hour of searching, tcpdumping, pfctl'ing, and swearing, I came to the following conclusions:
- the other machine, which could get connections to internet hosts, had the old router with the modem line as default gateway, and didn't go through the DSL link
- it seemed to be fast on web connections, because it used the web proxy on the DSL router for outgoing http requests
- even if I removed the special NAT rule for the iBook, I couldn't get out
- no state was created for any outgoing connection (in fact, tcpdump didn't show any outgoing packets, except those that originated from the local host)
...and I had forgotten to enable IP forwarding via the appropriate sysctl and the pf rules didn't have anything to do with the problem
NetCraft has posted this interesting news item (cache), which has a little background on information dissipation in the tcp reset problem case...
by using a "route add" with the wrong parameters, I today produced the following route on my MacOS X 10.2 box:
Destination Gateway Flags Refs Use Netif Expire
0&0xd5a443f1 194.121.21.23 UGSc 0 0 en0 =>
(...where 194.121.21.23 isn't even on the local net for that machine.)
Curiously, 0xd5a443f1 is 213.164.67.241 in the standard notation, but I don't know why MacOS suddenly chose to use a hex string in the route print output...
Strange.
Destination Gateway Flags Refs Use Netif Expire
0&0xd5a443f1 194.121.21.23 UGSc 0 0 en0 =>
(...where 194.121.21.23 isn't even on the local net for that machine.)
Curiously, 0xd5a443f1 is 213.164.67.241 in the standard notation, but I don't know why MacOS suddenly chose to use a hex string in the route print output...
Strange.
So now we know (cache) why everyone has been scrabmling to implement MD5 auth on their bgp peers in the past weeks...
Now, TCP reset attacks aren't exactly a new game (see here (cache) or even here (cache)), but Paul A. Watson (cache) presented why such an attack is feasible in real-world environments in his "Slipping in the Window: TCP Reset Attacks" CanSecWest '04 paper.
Apparently he shows that for an RST (or SYN) attack on a TCP connection, you don't need to hit the exact current sequence number, but merely a range defined by the window size & window scale parameters of the connection.
If the TCP stack additionally has a weak ISN generation scheme and a predictable algorithm for the selection of ephemeral ports, there's obviously a real threat for any kind of long-living TCP connection. And then, Cisco seems to use an "insanely huge" window size.
There's also this IETF draft (cache) with a description of the attack and possible defenses (didn't have the time to thoroughly read the text yet, though).
What makes this problem worse for BGP is the widely used practice of flap dampening (cache), which can result in routes from your AS being not globally routed anymore after only a few successful RST attacks on the BGP connection, but other core network protocols using TCP connections are also affected. MD5 auth helps, but at the cost of additional CPU usage, which may open an additional resource starvation attack vector on the router, so filtering transit networks used for bgp peers is probably the cheaper defense.
Another potential victim are VPN connections using TCP connections, as for example with OpenVPN or IPSEC over TCP for NAT traversal (or ssh sessions). There will shurely be a few more papers on this in the coming months...
Now, TCP reset attacks aren't exactly a new game (see here (cache) or even here (cache)), but Paul A. Watson (cache) presented why such an attack is feasible in real-world environments in his "Slipping in the Window: TCP Reset Attacks" CanSecWest '04 paper.
Apparently he shows that for an RST (or SYN) attack on a TCP connection, you don't need to hit the exact current sequence number, but merely a range defined by the window size & window scale parameters of the connection.
If the TCP stack additionally has a weak ISN generation scheme and a predictable algorithm for the selection of ephemeral ports, there's obviously a real threat for any kind of long-living TCP connection. And then, Cisco seems to use an "insanely huge" window size.
There's also this IETF draft (cache) with a description of the attack and possible defenses (didn't have the time to thoroughly read the text yet, though).
What makes this problem worse for BGP is the widely used practice of flap dampening (cache), which can result in routes from your AS being not globally routed anymore after only a few successful RST attacks on the BGP connection, but other core network protocols using TCP connections are also affected. MD5 auth helps, but at the cost of additional CPU usage, which may open an additional resource starvation attack vector on the router, so filtering transit networks used for bgp peers is probably the cheaper defense.
Another potential victim are VPN connections using TCP connections, as for example with OpenVPN or IPSEC over TCP for NAT traversal (or ssh sessions). There will shurely be a few more papers on this in the coming months...
The wireless world doesn't seem to get any better - here's (cache) the description of a tool called asleap (cache), which exploits weaknesses in Cisco's proprietary LEAP protocol (advertized as a "secure 802.1X EAP authentication solution").
It seems that Cisco tried to make shure that the successor to LEAP was widely used before this tool was made available, but didn't succeed. Their current recommendation (from last year, when the problems with LEAP first showed up, here (cache)) is to use better passwords. Hah.
I have no idea about how widely LEAP is used in the wild, but probably some companies bought into the idea after the WEP insecurities were public, and never upgraded their setup.
Also during the last week, Max Moser made available (cache) his tool HotSpotter (cache), which is able to lure Windows XP clients away from their EAP/TLS authenticated home network and make them connect to a rogue access point simulated by hotspotter.
Lots of new fun with "secure" wireless networks out there.
It seems that Cisco tried to make shure that the successor to LEAP was widely used before this tool was made available, but didn't succeed. Their current recommendation (from last year, when the problems with LEAP first showed up, here (cache)) is to use better passwords. Hah.
I have no idea about how widely LEAP is used in the wild, but probably some companies bought into the idea after the WEP insecurities were public, and never upgraded their setup.
Also during the last week, Max Moser made available (cache) his tool HotSpotter (cache), which is able to lure Windows XP clients away from their EAP/TLS authenticated home network and make them connect to a rogue access point simulated by hotspotter.
Lots of new fun with "secure" wireless networks out there.
Ausnahmsweise mal auf Deutsch hier...
Diesen Abend habe ich mich mal ernsthaft damit beschaeftigt, warum ich IPFilter nicht auf meiner Solaris Schuessel zum laufen bekomme, wodurch es zu folgender Unterhaltung auf dem IRC kam:
20:18 @Galaxis> Hm. Wie kann ich in nem binary nen memset call drin haben wenn
im source nichts in der Richtung vorkommt?
20:18 @double-p> bzero?
20:18 @Galaxis> Stimmt.
20:20 @Galaxis> Lustig. Entweder hat noch nie jemand ipfilter 4.1.1 unter
Solaris 9 compiliert oder man muss nur bei mir string.h dafuer
includen :)
20:21 @Galaxis> strings.h even
20:21 @double-p> Galaxis: :}
20:29 @Galaxis> Hm, strings.h ist auch bloedsinn, das ist das falsche bzero
20:31 @Galaxis> ne, ist ja nen kernel driver, da gibts bzero(9f)
20:31 @Galaxis> Vielleicht macht gcc und/oder der Linker was falsch
20:32 @double-p> oh
20:32 @Galaxis> (fuer bzero(9f) hat er die richtigen includes dabei)
20:32 @triple-n> bzero is builtin bei gcc afaicr
20:32 @Galaxis> Vielleicht sollte ich doch einfach nen binary package irgendwo
suchen :)
20:33 @triple-n> oder halt muehsam alle bzero()s durch memset()s ersetzen
20:33 @triple-n> wobei das stinkt, bzero() ist wunderschoen und jeder weiss
was es tut, memset hab ich oft genug mit vertauschten
parametern gesehen
20:34 @Galaxis> triple-n: solaris solaris solaris
20:34 @triple-n> galaxis, shouldn't matter, unix unix unix
20:35 @Galaxis> triple-n: Ja, aber solaris hat nen spezielles bzero fuer
kernel device driver, und da will ich natuerlich nicht das
builtin vom gcc haben
20:35 @triple-n> . o O (auf der hardware tut bestimmt auch OpenBSD... )
20:36 @triple-n> "spezielles bzero"
20:36 @triple-n> ha ha ha
20:36 @triple-n> interessant
20:36 @Galaxis> triple-n: aber nicht mit 2 CPUs
20:36 @triple-n> das braucht 4 CPUs um die memory pages mit random zeros zu
fuellen?
20:36 @triple-n> firewall mit 2 CPUs, sehr sinnig
20:36 @double-p> RANDOM ZERos
20:36 @double-p> MUHAAA
20:36 @triple-n> genau
20:36 @Elwood_G> nnn: der eine firet der andere walled \:)
20:37 @Galaxis> Wo ich das gerade sage faellt mir auf, dass ich es vielleicht
mal mit -fno-builtin versuchen sollte
20:37 @triple-n> *slap*
20:37 @double-p> gack
20:38 @Galaxis> triple-n: einmal ist das ne SS10 mit zwei SM51 (50MHz, yummi,
da verzichtet man nicht gerne auf die zweite CPU) und zweitens
geht es mir gerade nciht darum, irgendwas sinnvolles zu machen
:)
20:39 @triple-n> aeh, I see. da braucht solaris schon beide CPUs zum verwalten
der CPUs
*rotfl*
Siehe SolarisIpfilter :)
Diesen Abend habe ich mich mal ernsthaft damit beschaeftigt, warum ich IPFilter nicht auf meiner Solaris Schuessel zum laufen bekomme, wodurch es zu folgender Unterhaltung auf dem IRC kam:
20:18 @Galaxis> Hm. Wie kann ich in nem binary nen memset call drin haben wenn
im source nichts in der Richtung vorkommt?
20:18 @double-p> bzero?
20:18 @Galaxis> Stimmt.
20:20 @Galaxis> Lustig. Entweder hat noch nie jemand ipfilter 4.1.1 unter
Solaris 9 compiliert oder man muss nur bei mir string.h dafuer
includen :)
20:21 @Galaxis> strings.h even
20:21 @double-p> Galaxis: :}
20:29 @Galaxis> Hm, strings.h ist auch bloedsinn, das ist das falsche bzero
20:31 @Galaxis> ne, ist ja nen kernel driver, da gibts bzero(9f)
20:31 @Galaxis> Vielleicht macht gcc und/oder der Linker was falsch
20:32 @double-p> oh
20:32 @Galaxis> (fuer bzero(9f) hat er die richtigen includes dabei)
20:32 @triple-n> bzero is builtin bei gcc afaicr
20:32 @Galaxis> Vielleicht sollte ich doch einfach nen binary package irgendwo
suchen :)
20:33 @triple-n> oder halt muehsam alle bzero()s durch memset()s ersetzen
20:33 @triple-n> wobei das stinkt, bzero() ist wunderschoen und jeder weiss
was es tut, memset hab ich oft genug mit vertauschten
parametern gesehen
20:34 @Galaxis> triple-n: solaris solaris solaris
20:34 @triple-n> galaxis, shouldn't matter, unix unix unix
20:35 @Galaxis> triple-n: Ja, aber solaris hat nen spezielles bzero fuer
kernel device driver, und da will ich natuerlich nicht das
builtin vom gcc haben
20:35 @triple-n> . o O (auf der hardware tut bestimmt auch OpenBSD... )
20:36 @triple-n> "spezielles bzero"
20:36 @triple-n> ha ha ha
20:36 @triple-n> interessant
20:36 @Galaxis> triple-n: aber nicht mit 2 CPUs
20:36 @triple-n> das braucht 4 CPUs um die memory pages mit random zeros zu
fuellen?
20:36 @triple-n> firewall mit 2 CPUs, sehr sinnig
20:36 @double-p> RANDOM ZERos
20:36 @double-p> MUHAAA
20:36 @triple-n> genau
20:36 @Elwood_G> nnn: der eine firet der andere walled \:)
20:37 @Galaxis> Wo ich das gerade sage faellt mir auf, dass ich es vielleicht
mal mit -fno-builtin versuchen sollte
20:37 @triple-n> *slap*
20:37 @double-p> gack
20:38 @Galaxis> triple-n: einmal ist das ne SS10 mit zwei SM51 (50MHz, yummi,
da verzichtet man nicht gerne auf die zweite CPU) und zweitens
geht es mir gerade nciht darum, irgendwas sinnvolles zu machen
:)
20:39 @triple-n> aeh, I see. da braucht solaris schon beide CPUs zum verwalten
der CPUs
*rotfl*
Siehe SolarisIpfilter :)