SysAdmin Blog

SysAdmin Blog

Checkpoint vpn debug - Cannot signal vpnd: No such process

Alexander Bochmann Wednesday 05 of August, 2015
Jotting this down as I've found no useful reference to the above error message on the net:

Trying to enable IKE debugging on a Checkpoint FW1 using vpn debug ikeon results in the error message Cannot signal vpnd: No such process

This happens when the PID in in /opt/CPsuite-Rxx/fw1/tmp/vpnd.pid has been overwritten. I've seen various people (including myself) run into this because they had typed vpnd debug instead of vpn debug at some point...

Solution: Overwrite vpnd.pid with the correct PID

pgrep vpnd > $FWDIR/tmp/vpnd.pid

FortiOS 5.2 upgrade problems on Fortigate 80C

Alexander Bochmann Sunday 05 of July, 2015
Recently I tried upgrading my Fortigate 80C firewall to a current FortiOS (5.2.3) following the - supposedly - supported upgrade path, from 5.0.10.

Unfortunately I ran into the ehci_hcd 5035: fatal error that's been mentioned on the Fortinet forums in various places (here, for example) - system doesn't boot. Good thing it's possible to easily fall back to the previous release by booting from the backup partition. When you're connected to the console port, that is.

Today I found out FortiOS 5.2.3 can be installed after wiping the internal flash from the bootloader, using a serial console. My Fortigate had originally been installed with some FortiOS 4 release - I assume the boot disk layout has changed somewhere between releases, and the new image just doesn't fit.

  • a tftp server configured for an address in the network on interface 1 of your Fortigate to hold the new firmware image (mine wasn't, and I had to quickly shuffle some things around to recover from that)...
  • an USB stick with the current configuration to import after the upgrade has finished (or just put it on the tftp server, too)

First, select

[F]: Format boot device.

from the bootloader menu. As soon as that is finished, use

[G]: Get firmware image from TFTP server.

to fetch the new firmware image via tftp. The system will reboot with a default configuration. Log in with the admin account (no password) and restore your configuration from the USB stick:

config global
execute restore config usb <filename>


ATEN support - a positive surprise

Alexander Bochmann Friday 17 of October, 2014
In my previous post - quite some time ago - I was bitching about the scarcely documented RADIUS authorization function on one of our ATEN SN0116 serial console servers.

Some time later, I found out that RADIUS authentication doesn't quite work - only one user could log on when using RADIUS, and subsequent login attempts were denied. At the same time, there was no such problem when using local user accounts on the system.

Initially, I didn't bother opening a support request with ATEN for this issue, because I didn't expect anything to happen (no support contracts and all). Nevertheless, just before giving up on the RADIUS functionality, I registered one of our systems and described the problem.

After a bit of back and forth with the support representative on the ticket, I quickly got through both the "reporting this to engineering" and "engineering acknowledged the problem" stages, to get an updated software only a couple of days later.

It's not yet up on the ATEN firmware download page, but I assume the upcoming version (the next after v3.1.303) will have the relevant bug fix.

At our company, we've been quite exhausted by our support experiences with the likes of Cisco and Juniper in the past months, so this was really a pleasant experience for a change. Thank you, ATEN.

RADIUS authorization on an ATEN SN0116 serial console server

Alexander Bochmann Friday 07 of March, 2014
For several days I've been puzzled by the documentation for RADIUS authorization on an ATEN Altusen SN0116 serial console server (cache), using Windows NPS as RADIUS server. The ATEN docs unhelpfully state, "On the RADIUS server, set the access rights for each user according to the attribute information in the table, below" - and then there's a list of flags that specify the authorization options.

The docs fail to mention in which RADIUS attribute these authorization flags are supposed to be returned to the console server, though.

After some twiddling, it turns out that the flags should to be placed (in Microsoft NPS terms) as string into a vendor-specific attribute with vendor-code 0 and vendor-attribute 0. Additionally, if the RADIUS policy configuration contains several vendor-specific attributes, it seems that the ATEN device only parses the first one that's returned by the server.

NPS configuration to make this work looks something like this:

MS NPS RADIUS configuration for ATEN serial console server
MS NPS RADIUS configuration for ATEN serial console server

upgrading from php 5.2 to php 5.5 in one step...

Alexander Bochmann Friday 24 of January, 2014
...is actually not that hard (at least with something like my site, which is of limited complexity after all).

I ran into just a handfull of problems:
  • php does not start at all after Zend Opcache is enabled (php.ini with zend_extension=opcache.so, opcache.enable=1, etc.)
Fatal Error Unable to allocate shared memory segment of 134217728 bytes: shmat: Cannot allocate memory (12)

On my not-quite-current OpenBSD system, the kern.shminfo.shmmax sysctl was set rather low by default. For Opcache to work, it needs to be large enough to provide the amount of memory configured in opcache.memory_consumption (which is actually the value from the error message).
The one time fix is running sysctl -w kern.shminfo.shmmax=134217728, and otherwise to add the equivalent setting to sysctl.conf
  • error log is full of PHP Strict Standards and PHP Deprecated messages, and tweaking error_reporting in php.ini does not help at all
This actually cost me quite some time to fix until it finally clicked after I repeatedly read over "Prior to PHP 5.4.0 E_STRICT was not included within E_ALL" in the php documentation: Some of the (old) PHP applications I run set their own error reporting values, and one offender in particular had @error_reporting(E_ALL); in it's index.php... Changing that to @error_reporting(E_ALL ^ E_NOTICE ^ E_STRICT ^ E_DEPRECATED); keeps the log file clean (but I guess I'm not going to upgrade past php 5.5 anytime soon).
  • PHP Warning: strftime(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function.
That was unexpected, but easy to get rid of - obviously by actually setting date.timezone to a reasonable value in php.ini (CET in my case).

mout.web.de did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA

Alexander Bochmann Saturday 17 of August, 2013
Looks like some time on August 6, web.de changed something in the STARTTLS configuration of their mail system (possibly enabling it for the first time, as I don't see TLS info in headers of older mails), which was leading to SSL errors on my system, and mails were not being delivered:

Aug  6 14:17:49 sm-mta[20419]: STARTTLS=server, error: accept failed=0, SSL_error=1, errno=0, retry=-1
Aug  6 14:17:49 sm-mta[20419]: STARTTLS=server: 20419:error:1409442F:SSL routines:SSL3_READ_BYTES:tlsv1 alert insufficient security:/usr/src/lib/libssl/src/ssl/s3_pkt.c:1061:SSL alert number 71
Aug  6 14:17:49 sm-mta[20419]: r76CHng2020419: mout.web.de [] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA

Turns out other people ran into this problem too in the last couple of days, and there's a solution in this mail (cache): Pre-generating a DH parameters file for sendmail helps.

Use openssl to create a dhparms file:

openssl dhparam -out /etc/mail/sendmail.dh 1024

...then load it into sendmail by adding an confDH_PARAMETERS define to your .mc file:


...generate an updated sendmail configuration and restart the daemon.

decommissioned my old 3ware PCI IDE RAID controller last night...

Alexander Bochmann Thursday 04 of July, 2013
...after it did it's job for the better part of ten years, protecting my data from various disk failures, the last of which happened a couple of days ago:

Jun 27 20:40:22 kernel: [10918375.319638] 3w-xxxx: scsi0: AEN: WARNING: ATA UDMA downgrade: Port #3.
Jun 27 20:40:22 kernel: [10918375.592804] 3w-xxxx: scsi0: AEN: ERROR: Drive error: Port #3.
Jun 27 20:40:22 kernel: [10918375.597060] 3w-xxxx: scsi0: AEN: ERROR: Unit degraded: Unit #2.

It was a 5000 series card, one of the first affordable "real" IDE RAID controllers (unlike the other cheap ones that did most of the heavy lifting in the driver software). I last used it in the single PCI slot of an Intel Atom board, with two 1TB SATA drives connected via IDE-SATA bridge boards - a setup that worked, but with several drawbacks (like no S.M.A.R.T support, which was added only in later 3ware products):

3ware IDE RAID controller

I've currently replaced it with a pure software solution - using the mirroring feature of Linux's new btrfs filesystem on two SATA disks connected directly to the Atom board. Remains to be seen if that works as reliable as the old setup.

hardcoding AD site selection on a member server

Alexander Bochmann Thursday 14 of February, 2013
With Windows Server 2008, Microsoft has introduced a Read-Only Domain Controller (RODC) role that adds an additional layer of security to protect an Active Directory infrastructure from member systems that are located in an not completely trusted environment (say, a DMZ or perimeter network). Configured correctly, using an RODC can prevent unauthorized changes to the ADS from a compromised member.

Microsoft has documented the whole RODC design and all configuration steps extensively in the Read-Only Domain Controller Planning and Deployment Guide.

Let's assume we have done all that, and end up with an RODC living happily in his small firewalled DMZ, waiting for other AD systems to use his services. We move a server from the main network to the same DMZ, assuming everything will just work. And then it doesn't:

Active Directory has the nifty concept of AD sites, which allows clients and member servers to find appropriate Domain Controllers that are assigned to a specific location, using DNS. As detailed in the Planning and Deployment Guide, an RODC will be placed in it's own site, with appropriate AD site links to allow access to specific writable DCs (and the appropriate firewall rules to permit just that communication).
A member server moved to the RODC DMZ will not be able to contact any of the Domain Controllers it knows of from it's old site though, which causes the usual site selection mechanism to fail. With no knowledge of the site membership, our server is unable to find an assigned RODC, and all AD services will fail.

So we need a way to bootstrap site selection without AD, and lo - Microsoft has provided a way to do just that: There's a Registry key to tell an AD member which site it is in. Unfortunately the information is somewhat hidden in a subsection of the Deploying RODCs in the Perimeter Network document...

The Registry entry is:

HKLM\System\CurrentControlSet\Services\Netlogon\Parameters\SiteName (string)

The value of that key is the actual site name.

With that information, a DMZ system can create the appropriate DNS requests to find the DCs for it's site, and most things will be ok.
One of the things that needs additional work is DNS registration, though: As an Read-Only DC doesn't have the permission to update DNS entries through Active Directory, systems in an RODC site will need to send DNS updates to a writable Domain Controller. Microsoft has documented that part in DNS updates for clients that are located in an RODC site.

renumbering the external interface on a Juniper SA 4500

Alexander Bochmann Saturday 01 of December, 2012

1. Intro

Moving the external interface of a Juniper SA 4500 SSL-VPN appliance to a different IP subnet is not an entirely illogical procedure, but due to the many interdependencies between configuration elements, the exact workflow is somewhat non-obvious... It's also not mentioned in the official system documentation. In this case, I was working on an SA 4500 active/passive - cluster running R7.1 sofware.

The main problem is that all IP addresses on the external interface have to be from the same subnet - and it's not possible to define additional VLANs with other address ranges on the external side. It's also not possible to move any part of the external interface configuration to a new subnet as long as there's still configuration from the old address range somewhere on the system.

2. Useful preparation steps

  1. Configuration backup: Archiving -> Local Backups -> Save Configuration
  2. Sort of a migration plan: A mapping from old to new external IP addresses on the system, and preparation for the required DNS changes. You'll need Netmask and Default Gateway of your external port, external IP addresses from all cluster members, the external cluster VIP, and any addresses used on external virtual ports.
  3. As long as you are only using SSL transport (and not IPSEC), having a NAT gateway in front of the SA is also a great help - just do destination NAT from old to new addresses, and DNS changes can be postponed to any time later on without further interruption of service.

3. Sequence of configuration changes to remove all external IP addresses

  1. Configuration -> Certificates -> Device Certificates has a list of SSL certs for all hostnames on the appliance. Each Certificate is bound to one or more Internal or External Virtual Ports. Remove all Selected Virtual Ports from any Certificate that's bound to an External Virtual Port.
  2. Network -> External Port -> Virtual Ports shows all the additional external addresses configured on the SSL VPN gateway. Select "Settings for: Entire Cluster" and delete all external Virtual Ports.
  3. Now that all External Virtual Ports are gone, it's possible to disable the External Interface entirely (in fact, it can be disabled any time, but then the Virtual Ports are locked and can't be deleted): Go to Network -> External Port -> Settings, make shure to select Settings For: Entire Cluster and change Use Port? to Disabled. When navigating away from the page, note that the UI likes to jump to the configuration of the active cluster member, and Settings For: Entire Cluster has to be reselected from the Network Settings menu.
  4. On the disabled port, all IP configuration can be removed - just delete Netmask and Default Gateway for the Cluster.
  5. Delete IP Addresses for each cluster member: Network -> External Port -> Settings, switch the view between all nodes using the Settings For: - dropdown
  6. Remove the External VIP from Clustering -> Properties

4. Reenable networking on the external interface

With this, all external IP configuration is removed, and it's possible to reconfigure everything, starting from cluster settings for the external interface:

  1. Network -> External Port -> Settings, choose Settings For: Entire Cluster, enter Default Gateway and Netmask (which defines the subnet used on the external port)
  2. Switch through the cluster nodes and add an external IP Address from that subnet on each member
  3. Clustering -> Properties, add a new External VIP from the same network
  4. Reenable networking on the external Port: Network -> External Port -> Settings, Use Port? Enabled
  5. Add all the new Virtual Ports under Network -> External Port -> Virtual Ports
  6. Restore the binding between SSL certificates and your Virtual Ports on the Configuration -> Certificates -> Device Certificates page.

That's it for the SSL-VPN configuration - now you just need to make shure your destination NAT is in place (or that all DNS entries for addresses on your gateway have been changed to the new addresses).