SysAdmin Blog

SysAdmin Blog

mout.web.de did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA

Alexander Bochmann Saturday 17 of August, 2013
Looks like some time on August 6, web.de changed something in the STARTTLS configuration of their mail system (possibly enabling it for the first time, as I don't see TLS info in headers of older mails), which was leading to SSL errors on my system, and mails were not being delivered:

Aug  6 14:17:49 sm-mta[20419]: STARTTLS=server, error: accept failed=0, SSL_error=1, errno=0, retry=-1
Aug  6 14:17:49 sm-mta[20419]: STARTTLS=server: 20419:error:1409442F:SSL routines:SSL3_READ_BYTES:tlsv1 alert insufficient security:/usr/src/lib/libssl/src/ssl/s3_pkt.c:1061:SSL alert number 71
Aug  6 14:17:49 sm-mta[20419]: r76CHng2020419: mout.web.de [] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA

Turns out other people ran into this problem too in the last couple of days, and there's a solution in this mail (cache): Pre-generating a DH parameters file for sendmail helps.

Use openssl to create a dhparms file:

openssl dhparam -out /etc/mail/sendmail.dh 1024

...then load it into sendmail by adding an confDH_PARAMETERS define to your .mc file:


...generate an updated sendmail configuration and restart the daemon.

decommissioned my old 3ware PCI IDE RAID controller last night...

Alexander Bochmann Thursday 04 of July, 2013
...after it did it's job for the better part of ten years, protecting my data from various disk failures, the last of which happened a couple of days ago:

Jun 27 20:40:22 kernel: [10918375.319638] 3w-xxxx: scsi0: AEN: WARNING: ATA UDMA downgrade: Port #3.
Jun 27 20:40:22 kernel: [10918375.592804] 3w-xxxx: scsi0: AEN: ERROR: Drive error: Port #3.
Jun 27 20:40:22 kernel: [10918375.597060] 3w-xxxx: scsi0: AEN: ERROR: Unit degraded: Unit #2.

It was a 5000 series card, one of the first affordable "real" IDE RAID controllers (unlike the other cheap ones that did most of the heavy lifting in the driver software). I last used it in the single PCI slot of an Intel Atom board, with two 1TB SATA drives connected via IDE-SATA bridge boards - a setup that worked, but with several drawbacks (like no S.M.A.R.T support, which was added only in later 3ware products):

3ware IDE RAID controller

I've currently replaced it with a pure software solution - using the mirroring feature of Linux's new btrfs filesystem on two SATA disks connected directly to the Atom board. Remains to be seen if that works as reliable as the old setup.

hardcoding AD site selection on a member server

Alexander Bochmann Thursday 14 of February, 2013
With Windows Server 2008, Microsoft has introduced a Read-Only Domain Controller (RODC) role that adds an additional layer of security to protect an Active Directory infrastructure from member systems that are located in an not completely trusted environment (say, a DMZ or perimeter network). Configured correctly, using an RODC can prevent unauthorized changes to the ADS from a compromised member.

Microsoft has documented the whole RODC design and all configuration steps extensively in the Read-Only Domain Controller Planning and Deployment Guide.

Let's assume we have done all that, and end up with an RODC living happily in his small firewalled DMZ, waiting for other AD systems to use his services. We move a server from the main network to the same DMZ, assuming everything will just work. And then it doesn't:

Active Directory has the nifty concept of AD sites, which allows clients and member servers to find appropriate Domain Controllers that are assigned to a specific location, using DNS. As detailed in the Planning and Deployment Guide, an RODC will be placed in it's own site, with appropriate AD site links to allow access to specific writable DCs (and the appropriate firewall rules to permit just that communication).
A member server moved to the RODC DMZ will not be able to contact any of the Domain Controllers it knows of from it's old site though, which causes the usual site selection mechanism to fail. With no knowledge of the site membership, our server is unable to find an assigned RODC, and all AD services will fail.

So we need a way to bootstrap site selection without AD, and lo - Microsoft has provided a way to do just that: There's a Registry key to tell an AD member which site it is in. Unfortunately the information is somewhat hidden in a subsection of the Deploying RODCs in the Perimeter Network document...

The Registry entry is:

HKLM\System\CurrentControlSet\Services\Netlogon\Parameters\SiteName (string)

The value of that key is the actual site name.

With that information, a DMZ system can create the appropriate DNS requests to find the DCs for it's site, and most things will be ok.
One of the things that needs additional work is DNS registration, though: As an Read-Only DC doesn't have the permission to update DNS entries through Active Directory, systems in an RODC site will need to send DNS updates to a writable Domain Controller. Microsoft has documented that part in DNS updates for clients that are located in an RODC site.

renumbering the external interface on a Juniper SA 4500

Alexander Bochmann Saturday 01 of December, 2012

1. Intro

Moving the external interface of a Juniper SA 4500 SSL-VPN appliance to a different IP subnet is not an entirely illogical procedure, but due to the many interdependencies between configuration elements, the exact workflow is somewhat non-obvious... It's also not mentioned in the official system documentation. In this case, I was working on an SA 4500 active/passive - cluster running R7.1 sofware.

The main problem is that all IP addresses on the external interface have to be from the same subnet - and it's not possible to define additional VLANs with other address ranges on the external side. It's also not possible to move any part of the external interface configuration to a new subnet as long as there's still configuration from the old address range somewhere on the system.

2. Useful preparation steps

  1. Configuration backup: Archiving -> Local Backups -> Save Configuration
  2. Sort of a migration plan: A mapping from old to new external IP addresses on the system, and preparation for the required DNS changes. You'll need Netmask and Default Gateway of your external port, external IP addresses from all cluster members, the external cluster VIP, and any addresses used on external virtual ports.
  3. As long as you are only using SSL transport (and not IPSEC), having a NAT gateway in front of the SA is also a great help - just do destination NAT from old to new addresses, and DNS changes can be postponed to any time later on without further interruption of service.

3. Sequence of configuration changes to remove all external IP addresses

  1. Configuration -> Certificates -> Device Certificates has a list of SSL certs for all hostnames on the appliance. Each Certificate is bound to one or more Internal or External Virtual Ports. Remove all Selected Virtual Ports from any Certificate that's bound to an External Virtual Port.
  2. Network -> External Port -> Virtual Ports shows all the additional external addresses configured on the SSL VPN gateway. Select "Settings for: Entire Cluster" and delete all external Virtual Ports.
  3. Now that all External Virtual Ports are gone, it's possible to disable the External Interface entirely (in fact, it can be disabled any time, but then the Virtual Ports are locked and can't be deleted): Go to Network -> External Port -> Settings, make shure to select Settings For: Entire Cluster and change Use Port? to Disabled. When navigating away from the page, note that the UI likes to jump to the configuration of the active cluster member, and Settings For: Entire Cluster has to be reselected from the Network Settings menu.
  4. On the disabled port, all IP configuration can be removed - just delete Netmask and Default Gateway for the Cluster.
  5. Delete IP Addresses for each cluster member: Network -> External Port -> Settings, switch the view between all nodes using the Settings For: - dropdown
  6. Remove the External VIP from Clustering -> Properties

4. Reenable networking on the external interface

With this, all external IP configuration is removed, and it's possible to reconfigure everything, starting from cluster settings for the external interface:

  1. Network -> External Port -> Settings, choose Settings For: Entire Cluster, enter Default Gateway and Netmask (which defines the subnet used on the external port)
  2. Switch through the cluster nodes and add an external IP Address from that subnet on each member
  3. Clustering -> Properties, add a new External VIP from the same network
  4. Reenable networking on the external Port: Network -> External Port -> Settings, Use Port? Enabled
  5. Add all the new Virtual Ports under Network -> External Port -> Virtual Ports
  6. Restore the binding between SSL certificates and your Virtual Ports on the Configuration -> Certificates -> Device Certificates page.

That's it for the SSL-VPN configuration - now you just need to make shure your destination NAT is in place (or that all DNS entries for addresses on your gateway have been changed to the new addresses).

jloader-ex-2200 - Error: upgrade address wrong

Alexander Bochmann Saturday 03 of March, 2012
Just in case anyone else gets hit by this (increasingly unlikely, as it's old software):

Tried to upgrade two Juniper EX2200 switches running JunOS 10.2R1.8 with the new bootloader to support dual-root firmware, and ran into the following error:

Verified jloader-ex-2200-11.3I20110326_0802_hmerge.tgz signed by PackageDevelopment_11_3_0
Adding jloader-ex-2200...
Installation in progress, please wait...
Mounted jloader-ex-2200 package on /dev/md10...
Registering jloader-ex-2200 as unsupported
Error: upgrade address wrong 0x180000 0x100000 0x700000
Installation failed with exit status 1
Saving package file in /var/sw/pkg/jloader-ex-2200-11.3I20110326_0802_hmerge-signed.tgz ...
Saving state for rollback ...

The only place where this message is mentioned is in this post on the J-Net forums (cache), but I couldn't really get how things worked out for the author from that thread.

So here's what I found when upgrading my systems:

Jloader will in fact not be installed after this error. JunOS has to be upgraded first (10.4R8.5 in my case). It will still boot with the old boot loader (just dual-root support won't work).
After a reboot, the jloader package has to be downloaded and installed a second time with the usual request system software add .... With JunOS 10.4 as a base, the "upgrade address wrong" message will disappear. Reboot again, then remove the jloader installer with request system software delete jloader-ex-2200 to get rid of the "At least one package installed on this device has limited support" - warning. Reboot a third time.

This needs quite some time on EX2200 (CPU on these boxes is really, really slow) - maybe 45 minutes all in all - but works without further problems in the end.

apt-get: Reinstallation of [package] is not possible, it cannot be downloaded

Alexander Bochmann Sunday 15 of January, 2012
It's implicitly in the apt-get manpage, but I didn't see it until now - downgrading Ubuntu/Debian packages (for example to remove packages from an Ubuntu-PPA or a manually installed .deb with a newer version): Just add /distribution-name to the package when reinstalling.

For example (Ubuntu):

apt-get install --reinstall libgnutls26/oneiric libgcrypt11/oneiric

Xorg: DPI set to (20, 20)

Alexander Bochmann Thursday 12 of January, 2012
So I have this rather old Celeron-300 laptop that I install a new version of Ubuntu on from time to time to see if it still works.

It's had the problem that DPI gets set to a wrong value (20) and consequently I get a really tiny font for quite some time, and after the old way of forcing --dpi on the X comand line doesn't seem to work anymore, it was finally time to find out why.

Turns out the newly created /etc/X11/xorg.conf contains garbage in the form of a wrong DisplaySize and PanelSize:
wrong DisplaySize in Xorg.conf
Section "Monitor"
        Identifier   "Monitor0"
        VendorName   "Monitor Vendor"
        ModelName    "Monitor Model"
        DisplaySize  800 600

Both contain the resolution of the display in pixels in the autogenerated xorg.conf, but are supposed to have the size of the panel in millimetres instead.

After setting DisplaySize to the correct values (and just commenting out PanelSize in the Device section), the resulting DPI looks much more sane...

DisplaySize 245 185

...creates the following DPI value:

(II) SMI(0): Output LVDS using initial mode 800x600
(**) SMI(0): Display dimensions: (245, 185) mm
(**) SMI(0): DPI set to (82, 82)

netdot vs. Alcatel OmniSwitch and Juniper EX

Alexander Bochmann Sunday 08 of January, 2012

1. intro

In december, I stumbled over a short presentation (in german) on netdot and RANCID by Jens Link (cache). I've been a fan of RANCID for a long time, but never had heard of netdot, the Network Documentation Tool (cache).

Turns out netdot does a lot of useful things via SNMP::Info (cache), from pulling inventory data to reading out VLAN, IP, ARP and forwarding tables, and then puts all that in a neat web interface for further classification. It's possible to document just about everything around network systems in netdot, down to the cabling setup.
On the logical side, netdot can be used for IP address and VLAN management, and to create configurations files for RANCID, ISC bind and dhcpd, and Nagios.

All in all it seems like a pretty nifty tool, and I've only scratched the surface of it's functionality until now - just having the MAC database (and by that the information where each end system is attached) makes setting up netdot worthwhile.

2. basic setup

My installation is based on netdot 0.9.10 on a Debian Squeeze LAMP system.
Setup is quite straightforward, just follwing the available documentation: Dowload, unpack, run make installdeps-apt-get to install all missing packages, run make installdeps to fetch anything that might still be missing from CPAN.
Then copy etc/Default.conf to etc/Site.conf and adapt all settings for your installation, especially those for the database connection. A make installdb will take care of setting up the DB. (I'm not very fond of the way that's done, as the routine needs database admin credentials and fails if you create a database in advance with a user that just has admin rights on that.)
After that, make install copies everything to /usr/local/netdot. I've used checkinstall to create a basic debian package:
checkinstall --fstrans=no env APACHEUSER=www-data APACHEGROUP=www-data make install

As a last step, netdot needs to be added to the apache configuration, by copying a template: cp /usr/local/netdot/etc/netdot_apache2_local.conf /etc/apache2/conf.d/

3. installation woes

At this point, everything works in principle, including logging in to the web interface. Unfortunately, that's also where the problems start...

3.1. Can't add any system to the database

updatedevice.html produced the following error: 
Device::_get_snmp_session: Cannot connect to <Device>. Tried communities: public

Looking at network traffic with tcpdump, no snmp queries go out at all... After tinkering with SNMP::Info internals for two hours, found the unexpected solution in the netdot bugtracker, bug #1622 (cache):
Two debian packages are missing in the installdeps phase: apt-get install snmp-mibs-downloader smistrip

3.2. Alcatel MIBs missing

netdot contains a collection of MIB files that's probably derived from the netdisco-mibs package. When trying to add Alcatel OmniSwitch systems on our network, SNMP::Info barfs - it has an AlcatelLucent.pm to handle those systems, but that doesn't work without the MIBs. In the end I found a relatively complete set of OmniSwitch MIBs in the Observium SVN (cache) (because, you know, those files are so secret that you can't download them from the Alcatel-Lucent web site - I'll be so happy when we're rid of the last of these things).
To make things somewhat more fun, Alcatel provides a customized AaIETF_HUBMIB_POWER_ETHERNET_DRAFT.mib that uses the same module name as the official POWER-ETHERNET-MIB. SNMP::Info::Layer3::AlcatelLucent documentation advises to change it's module name to ALU-POWER-ETHERNET-MIB by editing the file. Which is fine, except that the corresponding error also comes up when there's another problem (like a dependant mib still missing).

3.3. Juniper EX non-obviously not supported

Adding our Juniper EX 4200 and EX 4500 switches to the database seemed to work fine at first, until it turned out that just the data for the system that was added last is in the database.
I did not have a look at netdot's data model, but obviously the problem was caused by IP addresses used on internal interfaces that are identical on all EX systems (128.0.0.x). Fortunately, Site.conf provides a configuration directive to weed out these pseudo interfaces:
-IFRESERVED  => "pimd|vt-|tap|pe-|pd-|dsc|rptr|unrouted",
+# bme,pime,mtun,lsi,jsrv: Juniper
+# internal (interface): Alcatel
+IFRESERVED  => "bme|pime|mtun|lsi|jsrv|pimd|vt-|tap|pe-|pd-|dsc|rptr|unrouted|internal",

Next I noticed that the VLAN information gathered from Juniper switches was broken: Instead of the VLAN ID, the database just contained sequence numbers.
Obviously EX switches use a different method to provide VLAN information via SNMP than Juniper routers. Someone has already provided an JuniperEX.pm extension to SNMP::Info, but it's not integrated into the distribution yet.
I ended up reinstalling a current version of SNMP::Info directly from CPAN, and then just copy the JuniperEX.pm file into /usr/local/share/perl/5.10.1/SNMP/Info/Layer3/ (path on my system) and patch SNMP/Info.pm:
--- Info.pm.orig        2012-01-07 15:10:27.000000000 +0100
+++ Info.pm     2012-01-07 15:13:00.000000000 +0100
@@ -1349,6 +1349,10 @@
         $objtype = 'SNMP::Info::Layer2::HPVC'
             if ( $desc =~ /HP\sVC\s/ );

+       # Juniper EX switch and SRX routers
+       $objtype = 'SNMP::Info::Layer3::JuniperEX'
+           if $desc =~ /\b(ex[234][25]00-\w+|srx\d+\w+)\b/;
         # Generic device classification based upon sysObjectID
         if (    ( $objtype eq 'SNMP::Info::Layer3' )
             and ( defined($id) )

...except the netdot MIB package doesn't contain JUNIPER-VLAN-MIB and it's dependencies, but that can be solved easily by downloading a complete collection of Juniper MIBs (cache) from juniper.net.

4. unsolved problem

After all that, my netdot installation happily crunches data from about 90 switches in our central location, and I'll most probably add our external sites in the coming weeks.
One last problem remains - SNMP::Info doesn't read inventory information from Juniper EX switches, so I'll have a look at exending the existing JuniperEX.pm. More news when that actually works.

At last, a thanks to the developers - got a detailed description on where to look when I posted my various problems to the netdot-users mailing list.

28c3, day 1

Alexander Bochmann Wednesday 28 of December, 2011
So it's that time of the year again...

Feels somewhat different from last time over (25C3, has it really been three years?):
  • Less crowded. Maybe just because it's day one, but most things run smoothly (exception: trying to get a r0ket, I think 50% of the attendands were in that queue). No probelm to find place with a free power socket.
  • Less laptops in the lectures. The desks in Saal1 have been removed, and I have the impression that the same happened to lots of wall sockets. Ok, people switched to tablets and smartphones, but even that seems at a reduced level.

I'm not going to write summaries of speeches, but I noticed that the topic of (OSI) layer boundary violations came up twice - once in the session on PHY attacks on wireless devices, and when Dan Kaminsky mentioned something in the line of "applications built on top of TCP assume reliability" (and that assumption can be broken). It seems there's some surprises still waiting in that direction.

Cory Doctorow was good for a host of quotes, but I was hit by his description of networked activism, and why it seems so disparate without being desolate, which went something like this: Earlier on, the main work was actually organizing the activist movement, and so people agreed on common goals first and then went on organizing around that ("2% figuring out what to do, 98% stuffing envelopes"). Nowadays, organization comes mostly for free, and it's easy to form up around a topic, start discussing things, and maybe move off in different directions without losing touch. Which is exactly what confuses people about the pirate party in germany, for example (or Anonymous, or Occupy). Now that maybe oversimplified, but it rings true in principle.

His talk is here on youtube: http://youtu.be/OyNmUmmQ0kI

Nebenbei scheinen in Sachsen manche Dinge anders zu funktionieren als anderswo: http://youtu.be/Cz22Vx1MRzM

Checkpoint SmartView Monitor R70.x script error

Alexander Bochmann Sunday 11 of December, 2011
I've now seen several installations of an Checkpoint R70 management GUI where the details pane in SmartView Monitor doesn't work because of Javascript errors like "'statSection' is undefined" or similar. The URL usually is "file:///C:/Program%20Files/CheckPoint/SmartConsole/R70.40/PROGRAM/data/htdocs/overview.html?", as in the following screenshot (german version):

checkpoint SmartView Monitor script error

I've found several references to this problem on the Checkpoint support forums and elsewhere, but none with a conclusive solution.

Obviously, this error occurs because SmartView Monitor can't find several files in PROGRAM/data/htdocs.

I have a somewhat convoluted workaround for this:
  1. There are two .tar files in PROGRAM/data which contain the required files, namely SmartViewMonitor.tar and SmartViewTracker.tar.
  2. For some reason it's impossible to untar those two files directly in the data folder, even as Administrator - I have the impression that a htdocs directory already exists, but is invisible or has very strange access restrictions (have found no way to make it visible).
  3. So copy the two files to somewhere else, and untar them using either an archiver like 7-Zip, or with the gtar.exe that's included in PROGRAM/util with any SmartDashboard installation.
  4. Move the resulting htdocs folder back to PROGRAM/data within the Checkpoint GUI directory. You'll need administrative privileges, and force overwriting any existing files.
  5. Never see script errors again.