SysAdmin Blog

The iPXE open source boot firmware project provides an CD image that boots the iPXE binary using isolinux.

Over on the Fediverse, the topic of bootstraping a system from a floppy disk came up, and with the iPXE binary being a mere 330KB, there's really no reason why it shouldn't be possible to boot that from a floppy disk. And it actually does work, with a few simple steps (on a Debian-ish Linux):

• format floppy disk and create FAT filesystem

  fdformat /dev/fd0
  mkfs -t fat /dev/fd0

• get syslinux and install to floppy

  apt install syslinux syslinux-utils
  syslinux --install /dev/fd0

• get iPXE ISO

  curl -O http://boot.ipxe.org/ipxe.iso

• mount both iPXE ISO and floppy, copy over required files, rename isolinux.cfg to syslinux.cfg

  mkdir fd iso
  mount /dev/fd0 fd
  mount -o ro ipxe.iso iso
  cp iso/ipxe.krn fd/
  cp iso/boot.cat fd/
  cp iso/isolinux.cfg fd/syslinux.cfg
  umount fd
  umount iso
  rmdir fd iso
  

That's all! Take your floppy and boot a system

Once iPXE has been started, hit Ctrl-B to call the shell. If you have a DHCP server on your network and a web server with a bootable ISO image, it's just two iPXE commands:

dhcp
sanboot http://<webserver>/<filename>.iso

We've been playing with a SolidFire flash storage cluster for some time, and recently wanted to update the nodes to the current ElementOS 10.1 release.

Unfortunately, our FDVA management node installation was borked, so we decided to just roll a new one from the current VM appliance template - easy.
As it turns out though, the FDVA appliance only ships with the latest software release files, and the individual SolidFire nodes check back for a repository with their current version before starting the update, which consequently fails (it's all very Ubuntu-ish):

admin@SF-7323:~$ sudo sfinstall 192.168.10.21 -u admin -p password -l
2017-12-20 17:27:52: sfinstall Release Version: 10.1.0.83 Revision:  Build date: 2017-11-23 01:27
2017-12-20 17:27:52: Checking connectivity to MVIP 192.168.10.21
2017-12-20 17:27:52: Successfully connected to cluster MVIP
2017-12-20 17:27:53: PrintRepositoryPackages failed: SolidFireApiError server=[192.168.10.10] method=[AptUpdate], params=[{'quiet': 2}] - error name=[xCheckFailure], 
message=[cmdResult={ rc=255 stdout="W: Failed to fetch http://192.168.10.10/fluorine-updates/ubuntu/dists/precise/main/binary-amd64/Packages  404  Not Found
[..]
W: Failed to fetch http://192.168.10.10/fluorine-updates/security-ubuntu/dists/precise-security/universe/binary-amd64/Packages  404  Not Found

The SolidFire docs don't really mention what to do from there, so we tinkered around for some time and found this:

Any older version of the repository can be fetched using the update-fdva tool with the currently used ElementOS release version as command line (version number can be seen on the cluster web UI or when asking the cluster nodes for their mnode repository using sfinstall). In our case, the active version was 9.2.0.43 -

admin@SF-7323:~$ sudo update-fdva 9.2.0.43
Get: 1 http://localhost precise Release.gpg [490 B]
Get: 2 http://localhost precise-updates Release.gpg [490 B]
[..]

This will fetch the 9.2.0.43 version of the SolidFire repository, but will also downgrade to the matching (old) versions of solidfire-fdva-tools and solidfire-python-framework...

admin@SF-7323:~$ dpkg -l | grep fdva
ii  solidfire-fdva-tools-fluorine-patch2-9.2.0.43               9.2.0.43                          SolidFire FDVA Tools 9 [fluorine-patch2]

...so we immediately reinstalled the current versions, using update-fdva again, this time with the current release version number:

admin@SF-7323:~$ sudo update-fdva 10.1.0.83

With all that in place, we could just run the update routine using the usual sfinstall command.

After dist-upgrading a Debian system recently, I wondered which packages might have been left over from previous releases (the system in question has been through several dist-upgrades over its lifetime), even after running apt-get autoremove and deborphan. After dropping that question on Mastodon (cache), I got an answer pointing to apt-show-versions, which I didn't know of up to now.

This totally does what I've been looking for. From the man page:

NAME
       apt-show-versions - Lists available package versions with distribution

DESCRIPTION
       apt-show-versions parses the dpkg status file and the APT lists for the installed and available package
       versions and distribution and shows upgrade options within the specific distribution of the selected package.

       This is really useful if you have a mixed stable/testing environment and want to list all packages which are
       from testing and can be upgraded in testing.

Since I didn't have a package cache for apt-show-versions from the older release, all old packages are currently just shown with a No available version in archive comment. But since current packages are being tagged with the release, I can just exclude those with a simble grep:

# apt-show-versions | egrep -vc jessie
58

Another of those "just so I find my own post the next time I'm looking for this" things...

After the replacing the CPU fan on an old Intel D945GCLF2 Atom board, I "optimized" the BIOS settings by enabling automatic fan control (instead of having the fan at a fixed speed).

Currently, I have lots of messages like this in my kernel log, and a kworker thread using 100% CPU:

ACPI Error: Method parse/execution failed [\_SB_.PCI0.LPC_.SMBR] (Node ffff88007ec3f900), AE_AML_INFINITE_LOOP (20140424/psparse-536)
ACPI Error: Method parse/execution failed [\_SB_.PCI0.LPC_.INIT] (Node ffff88007ec3f928), AE_AML_INFINITE_LOOP (20140424/psparse-536)
ACPI Error: Method parse/execution failed [\_GPE._L00] (Node ffff88007ec35bd0), AE_AML_INFINITE_LOOP (20140424/psparse-536)
ACPI Exception: AE_AML_INFINITE_LOOP, while evaluating GPE method [_L00] (20140424/evgpe-580)

So, an hour or so of searching later, I finally hit this comment on the Novell bugzilla (cache) - and then I promptly remembered that I used to have known about this problem, and that it was exactly the reason why the fan was set to a fixed speed:

I've recently tried to reconfigure a mailinglist that I run on one of my systems to make less problems with recipients that use DMARC.

To that end, I wanted to implement the first option mentioned in the corresponding DMARC FAQ for mailinglist administrators (cache): Don't do any changes to the message body and headers potentially covered by a DKIM signature.

On the mailinglist configuration side, this means not adding a list tag to the message subject, and not changing the body with an additional header or footer. (Adding the usual RFC2369/RFC2919 list headers is no problem.)

After doing that it turned out that my setup, using sendmail, still changed the body on some messages, thanks to the automatic MIME autoconversion that sendmail does.

Getting rid of that actually required some digging into sendmail configuration and documentation:

Responsible for message delivery to local programs is the aptly named "prog" mailer. Unfortunately, as far as the m4 configuration statements are concerned, all the variables for this mailer are called "SHELL"... The default flags for the prog/shell mailer are hardcoded to "[eu9]", according to the README (though these defaults are changed in some of the OSTYPE definitions in the m4 macro collection):


The meaning of the individual flags is documented in the OP manual:



So it seems I want to get rid of the 9 in the LOCAL_SHELL_FLAGS, and replace it by a 8...

Adding the following two statements to my sendmail m4 configuration source does exactly that:

dnl # disable MIME-Autoconversion for prog mailer
MODIFY_MAILER_FLAGS(`SHELL', `-9')
MODIFY_MAILER_FLAGS(`SHELL', `+8')

On a headless Raspberry Pi running raspbian/jessie, the /var/log/messages file is filling up with entries like these:

 rsyslogd-2007: action 'action 17' suspended, next retry is [..date..] [ try http://www.rsyslog.com/e/2007 ] 

It seems this message is generated when rsyslogd isn't able to deliver syslog messages to one of the destinations in rsyslog.conf

In the case a raspbian, it's obviously the entry at the end of the config that tries to pipe messages to |/dev/xconsole - which doesn't exist on a system that doesn't run X11...

The messages disappear after commenting out or deleting the corresponding lines:

#daemon.*;mail.*;\
#       news.err;\
#       *.=debug;*.=info;\
#       *.=notice;*.=warn       |/dev/xconsole

I really should file a bug report for this...

I'm pretty certain I used to know this - but for the next time I'm putting this into a search engine and don't find it in the Splunk docs:

One of our data sources writes structured data into our Splunk installation which contains variable names with dashes - in this particular case, access-time

It's no problem using such a variable in a lot of Splunk operations, but it fails in an eval, as it will be interpreted as a mathematical operation (access minus time).

There's two options to work around that:

1. the one mentioned in the Splunk documentation: Put the variable name in single quotes, i.e. | eval newtime='access-time' - constant
2. the other one is to simply rename the variable before working on it: | rename access-time AS accesstime | eval newtime=accesstime - constant

Mostly as a reminder to myself when I'm looking to solve this kind of problem the next time: Since the March 22, 2017 version of the FortiClient VPN Android app kept crashing on my mobile (still running the last Cyanogenmod 13 snapshot) as soon as I tried to switch away to the launcher, I wanted to downgrade the app.

Unfortunately, there's no copy on apkmirror.com or F-Droid, and I don't know about any other reasonably trustworthy sources. I also already had removed and reinstalled the app, so recovering the old version on the phone didn't seem an option either.

Fortunately, I take TWRP backups now and then, so I tried looking at one of those. For once, having unencrypted backups turned out real convenient: A TWRP data.ext4.win file is just a tar.gz, so I was able recover the app/com.fortinet.forticlient_vpn-1/base.apk file (using 7Zip on Windows), and copy that over to my phone. After uninstalling the current version of the FortiClient app, I just reinstalled the program with the CM file manager using the restored base.apk as a source. Done.

Cisco ASA firewalls don't log, by default, the username used in a failed administrator login. Instead, the login is masked out using "*" characters:

%ASA-6-113005: AAA user authentication Rejected : reason = AAA failure : server = 10.1.1.1 : user = ***** : user IP = 192.168.0.10

The rationale is that users sometimes enter their password instead of the username, and the password will then end up in logs. As we're using two-factor authentication for admin logins, that doesn't apply to us.

That behaviour was actually tracked as a bug in Cisco's bug database (cache), and while the article mentions that a command was introduced to change this behaviour, the command itself isn't mentioned.

After some fiddling on the ASA command line I found this statement:

no logging hide username

The corresponding button in the ASDM GUI is in Device Management -> Logging -> Syslog Setup: "Hide username if its validity cannot be determined"

...until I had a look at the DNS server log, which showed errors contacting other servers via IPv6.

The hoster I'm using has a somewhat strange IPv6 setup where you get a /64 for your system, but the default gateway is just fe80::1 - when I originally set up the system, I put that into /etc/mygate whithout thinking much about it.

This initially was ok for quite some time, but it seems the default route vanished at some point. (In retrospect I don't quite understand why the setup ever worked at all, as the lo0 lookback interface has fe80::1 auto-assigned too...)

Then I remembered that fe80:: carries interface tags, since it exists on any IPv6-enabled interface, and the OS needs some way to decide which fe80:: it has to deal with right now.

Edited /etc/mygate accordingly, and things are back to normal (vio is OpenBSD's VirtIO network device driver, so my virtual ethernet device is vio0):

fe80::1%vio0