SysAdmin Blog

vSphere host profiles, "A specified parameter was not correct: portgroupName"

2024-08-30T19:48:00+00:00

It's not often you run into a VMware error message that has almost no search engine hits, which we managed to do recently...

When trying to apply an existing host profile to a new host added to a cluster, vSphere errored out:

A general system error occurred: Batch host remediation failed.
A specified parameter was not correct: portgroupName

We later learned that the same error would appear when trying to reapply the host profile to a machine that had been in the cluster for quite some time...

I'll spare you the details, but as it turns out this happened ... because we were configuring vmkernel ports on dvSwitch portgroups in the host profile, and I had changed one of those port group names.

Knowing that, the error message suddenly makes sense 🙄

So instead of changing back the names, we went forward and updated all our host profiles with the new designations, which also meant reacknowledging host profile customizations that were referencing these port groups.

force serial console on an HP apollo 715/50 workstation

2024-05-05T13:22:00+00:00

I made the error of setting the console path to graphics in the BOOT_ADMIN console of an old HP apollo 715/50 workstation with no monitor connected (or at least none that is able to detect the system's VGA signal).

On more recent HP 9000 hardware, it seems to be possible to reset the console path to serial by pressing the TOC button after powering on with no keyboard and monitor connected, but as the NetBSD/hppa FAQ (cache) sais, this has no effect on a 715.

As it turns out, there is another way though, and I haven't seen it documented anywhere: The 715/50 has a monitor selection switch for its onboard graphics adapter. It has one setting (both switches on SW1 down) that is labeled as 15" Color (Model 715/33 only) in the service manual.

With this setting, the system comes up with serial A as default with 9600/8/n/1, and it's possible to interrupt the boot process with , select to get into boot administration mode, and then change the console path back to serial from the BOOT_ADMIN> prompt:

PATH console rs232_a.9600.8.none
RESET

Windows 10 and WSL: Thousands of "HNS Container Networking" firewall rules

2024-05-02T12:40:00+00:00

My main Windows 10 PC, originally installed in 2018, recently has been having strange networking problems after powering on. For example, WSL would not start for minutes, and Wireguard took ages to activate.

I happened to find this general WSL troubleshooting article (cache) on the Microsoft knowledgebase, which, about half way down, mentions possible problems with "HNS Firewall rules" and has a Powershell oneliner to remove some of those rules.

No idea why this was the first thing I tried out of the many options on that page, but as it turns out, my system had over 12.000 HNS Container Networking rules:

PS C:\Users\bochmann> Get-NetFirewallRule -name "HNS Container Networking*" | measure | select Count
Count
-----
12580

This seemed like a problem since there's only about 300 other firewall rules, not to mention the command took quite some time to complete.

After testing on my notebook, which has a much more recent Windows install, it turns out that each reboot adds six of these rules, provided I shut down the system with a shutdown /s /t 0 instead of using the Windows menu? Which I usually do to force a "real" shutdown and thwart fast startup...

On the notebook, I just nuked all HNS firewall rules (not just those for UDP/53), to no apparent ill effect (needs to be run as Administrator):

wsl --shutdown
Get-NetFirewallRule -Name "HNS Container Networking*" | Remove-NetFirewallRule
hnsdiag delete all
Restart-Service -Force hns

...on the other PC, Powershell tells me that the command will be running for another four hours.

Now I only need to find out why this happens in the first place.

manually applying patches from GitHub

2023-07-09T21:35:00+00:00

I wasn't previously aware that you can take any commit ID on the GitHub web interface and just add .diff to the URL to get a plain context diff that can then be applied to code existing elsewhere with good old patch.

So it's not required to fiddle with git repos and forks and whatever to quickly apply a patch out of band (and then return to the upstream state later on with something like a git checkout --force ... that squashes all the local changes).

Case in point: It was not initially clear when the recent Mastodon patches would be applied to the Hometown fork, but .diffs from relevant commits on the Mastodon repo applied to the code on my disk with minimal fuzz. So it was possible to quickly get into a state where my version had the most important patches without breaking the connection to Hometown upstream, and after the security fixes had landed there, I just checked that version out over my local changes.

quick notes from installing OS/2 Warp 4 way too often

2023-06-18T19:27:00+00:00

I own an old Via EPIA board with a C3 CPU, and for some reason I thought casually installing OS/2 would be a good idea.

I used install media copies from WinWorld
- I have German language install media in original packaging, but turning up all the patch sets in German was too much effort, and a mixed-language OS is annoying
- for some reason, the updated partitioning tool from the OS/2 Warp 4.52 installer failed on the IDE-to-SDcard adapter I was using
- (after lots of tries I ditched that storage solution and used an actual IDE disk - somehow boot sector and partition table kept getting lost when using the SD adapter?)
- fdisk from OS/2 Warp 4 worked without problems?
in BIOS setup, configure "LBA" addressing scheme for the HDD
OS/2 Warp 4 install CD is not bootable, you need install floppies (and a floppy drive)
- the installer has no USB support, and USB floppy drives are not an option (even though elstel.org, linked below, claims that booting from an USB floppy should work)
- (maybe some Via BIOS bug or something?)
- downloaded patched install disks from elstel.org (cache) (those with Dani's IDE driver, last option in the list)
- note these are the install disks, you also need a boot disk (I, uh, don't remember which one I used?)
- also note that elstel.org links to patched bootable OS/2 Warp 4 install CDs (didn't try those)
since the CDROM isn't bootable, I ended up using an SCSI drive behind a LSI/NCR/Symbios Logic 53C810 PCI SCSI card
- there are many releases of the 53C810 driver, but symbios406.zip from os2site.com was the newest one that worked for me with the Warp 4 install disks (versions newer than 4.0.x will hang, older versions may report unknown firmware)
- the 53C810 driver doesn't fit on the first install disk
- do not delete unneeded driver files from the install disk, instead truncate them (also mentioned on elstel.org)
- copying additional drivers from the install disks will fail when files are missing (will updating snoop.lst help?)
do not use quick install, it will create a FAT partition (instead of HPFS)
2GB HPFS install partition is fine
the EPIA C3 board has a 10/100 Via Rhine II, drivers on os2site, copy to an empty disk to install when enabling the TCP/IP stack
Via Soundblaster emulation (when enabled in the BIOS) is a Soundblaster Pro
after installation, I used this patchset from archive.org (note installation order mentioned in the TEXT file that's an additional download)
- has FP17, TCPIP 4.3 and the MPTS updates, Java runtime (not JDK), Netscape Navigator, Scitech SNAP with the "free" code

Debian bullseyse / Devuan chimaera openssl minimum TLS version

2022-04-23T13:29:00+00:00

I recently spent way too much time trying to find out why my mail server wasn't able to send mail to a system that apparently only supported TLSv1. None of the TLS options in the sendmail configuration made any difference.

Things started to click only after I noticed that connecting to the system in question via openssl s_client produced the same error message:

> openssl s_client -connect mail.some.domain:25 -starttls smtp
CONNECTED(00000003)
139770261177664:error:1425F102:SSL routines:ssl_choose_client_version:unsupported 
protocol:../ssl/statem/statem_lib.c:1957:

As it turns out, /etc/ssl/openssl.cnf in current Debian / Devuan has the following global configuration settings:

[system_default_sect]
MinProtocol = TLSv1.2
CipherString = DEFAULT@SECLEVEL=2

So yeah, anything using openssl that doesn't explicitly override that configuration will not be able to make TLS connections to systems that don't support TLSv1.2...

Changing the settings to MinProtocol = TLSv1 made it possible to deliver my mail.

network interfaces renamed following Proxmox 7 upgrade

2021-11-24T21:32:00+00:00

After upgrading my standalone Proxmox host from PVE 6 to 7, the interface names were suddenly changed back from "predictable" to the old ethX names. The setup is Proxmox on Debian, so when I initially set up the system, I manually installed Debian 10 first and then added the Proxmox 6 repositories and packages.

After some debugging it turned out there was an old systemd network configuration file that prevented systemd-udevd from starting up correctly:

systemd-udevd[xxxx]: /etc/systemd/network/99-default.link: No valid settings found in the [Match] section, ignoring file. To match all interfaces, add OriginalName=* in the [Match] section.

I currently have no idea where the file /etc/systemd/network/99-default.link originated from (it doesn't have a package owner after the upgrade), but apparently it contains an invalid syntax for the systemd-udevd in Debian Bullseye. Removing the file solved the problem, and I'm now back to the interface names in the ifupdown2 configuration used by Proxmox (I rebooted the system to be sure it comes up in the right way now).

WireGuard on the OpenPandora

2021-05-02T19:30:00+00:00

introduction

WireGuard is a VPN system built on modern cryptography that provides for a comparatively simple setup and uses UDP as a transport, with moderate overhead. It "just works" for road warrior setups where one end doesn't have a stable address.

The OpenPandora (cache) is an ARM Linux pocket computer, first released around 2010, that uses an ancient OpenEmbedded Ångström as base OS, with an Linux 3.2 kernel that has quite a few device-specific modules that never were upstreamed.

A couple of weeks ago, I decided to try to combine the two, provided I wouldn't turn out as too much of an effort. With that in mind, I looked at the wireguard-go userspace implementation instead of attempting the make the WireGuard linux-compat kernel module build against the outdated OpenPandora kernel.

Setting up a tunnel requires two WireGuard components:

a WireGuard protocol implementation (like the kernel module or wireguard-go)
a version of wireguard-tools that is used to provide a configuration to WireGuard

As for wireguard-go, I made a short attempt at trying to build golang on the Pandora itself, but hit the "too much effort" barrier pretty quickly. Fortunately, golang now provides for cross-compiling to supported platforms - but the Pandora is not one of those: The Pandora OS (SuperZaxxon) is built with the outdated "softfp" ARM binary ABI, which is backwards-compatible with ARM CPUs that don't have floating-point hardware, but actually is capable to use vfp and NEON in the backend, if supported by the compiler. The workaround here is to crosscompile with ARMv5 as target architecture, which produces a pure software floating point executable (that also works on softfp by design).

cross-building wireguard-go

I built wireguard-go on a Debian Buster host, and since buster-backports only provides go1.14, I couldn't use the most recent version (which currently requires go1.16): Went with wireguard-go 0.0.20210212 instead.

After checking out or unpacking the sources, building a binary is a simple matter of running make with the appropriate environment parameters:

env GOOS=linux GOARCH=arm GOARM=5 make

Just copy the resulting wireguard-go over to /usr/local/bin on your Pandora and make it executable.

compiling wireguard-tools

wireguard-tools has only a small set of build dependencies, the most important of which unfortunately isn't even mentioned: On Linux, you need a copy of the kernel headers that roughly matches the destination kernel.

Turns out that SuperZaxxon only ships the include files for the initial kernel (2.6), but not those for the last available kernel build. Also Linux 2.6 apparently doesn't provide some required functions, so my first attempt failed.

I ended up downloading the latest 3.2 kernel sources from the OpenPandora git.

When I compile software on the Pandora, I usually first try to use the cdevtools PND - it has an older gcc, but is generally more leightweight than the other option (Code::Blocks). So I start cdevtools, make a src/wireguard directory, and then download and unpack both wireguard-tools and the Pandora kernel sources in there.

In the wireguard-tools directory, go to src/ and run something like this:

env CFLAGS="-I`pwd`/../../pandora-kernel-pandora-3.2-c4c68a4/include -Os -mtune=cortex-a8 -mcpu=cortex-a8 -mfpu=neon -mfloat-abi=softfp -pipe" make

...and then, to install the resulting programs below /usr/local:

sudo env PREFIX=/usr/local WITH_WGQUICK=yes WITH_SYSTEMDUNITS=no make install

Pandora caveats

SuperZaxxon does not autoload the tun module, so /dev/net/tun doesn't exist. (Ironically, it would be loaded if /dev/net/tun did exist and then something tried to access the device...)
wg-quick uses some fancy bash i/o redirection which requires /dev/fd. Which is not there on the Pandora either, but it's easy to create, since it's just a symlink to /proc/self/fd.
Do not use a VPN interface name that starts with "w" (like the default of wg0)! It triggers bugs in other scripts on the OpenPandora, for example loading of the WiFi firmware will fail after a resume from sleep.
Add /usr/local/bin to the PATH of root so the binaries are found in their directory.
A couple of the advanced wg-quick functions fail, mostly due to missing or outdated tools. One that I encountered was changing nameservers, but I assume anything the makes changes to the firewall configuration will be broken too. I did not try calling external commands from the wg-quick config file yet (which might serve as a workaround for some uses).
Basic setup of a v4 tunnel with several routes has been tested successfully.
IPv6 is completely untested.

I wrote a small wrapper script that creates a suitable environment for wg-quick invocation that's included as /usr/local/bin/wg-pandora in the tar file below:

#!/bin/sh

if [ `id -u` -ne "0" ]; then
  echo "[!] script needs to be run as root, use su oder sudo"
  exit 1
fi

if [ "$1" == "" ]; then
  echo "[!] please use the VPN interface name as parameter"
  echo "NOTE: do not use any device names starting with \"w...\" -"
  echo "      it will prevent Wifi reconfiguration on SuperZaxxon."
  exit 1
fi

if [ ! -f /etc/wireguard/$1.conf ]; then
  echo "[!] please create /etc/wireguard/$1.conf with a valid wg-quick configuration"
  exit 1
fi

if [ ! -e /dev/net/tun ]; then
  echo "[+] load tun kernel module"
  modprobe tun
fi

if [ ! -e /dev/fd ]; then
  echo "[+] create missing /dev/fd symlink"
  ln -s /proc/self/fd /dev/fd
fi

echo "[+] launching wg-quick"
/usr/local/bin/wg-quick up "$1"

exit 0

installation

Download wireguard-pandora-20210502.tar.gz and unpack to the root directory:
```
tar -C/ -xpf wireguard-pandora-20210502.tar.gz
```
Create a wg-quick (cache) configuration in /etc/wireguard (man pages are included in the download, but man is not installed on the Pandora by default).
Run /usr/local/bin/wg-pandora . (Remember the note about interface names.)
You will need an existing WireGuard endpoint to connect to ;)
Manual setup using wg (see WireGuard quickstart) is also possible, as soon as the tun module has been loaded and wireguard-go is running.
There's a discussion thread over on the OpenPandora forums.

Apache httpd, reverse proxy, and caching

2020-11-24T21:24:00+00:00

There's tons of guides out there on either how to set up Apache httpd as a reverse proxy, or how to enable (disk) caching for content being served.

The web has surprisingly little information on how to combine both in a working manner, and to have Apache cache content that's being retrieved from a proxied backend.

Just using the default configuration and then dropping something like a CacheEnable disk into the that holds your proxy rules will not work: Nothing ever is written to the cache directory.

With debug logging you see either nothing at all or maybe a quick succession of AH00750: Adding CACHE_SAVE filter .. and AH00751: Adding CACHE_REMOVE_URL filter ... messages in the error.log

So what's up? Likely your configuration is entirely correct, but you're missing one statement:

CacheQuickHandler off

It seems that with the default of CacheQuickHandler being enabled, proxied content never hits the quick handler phase that allows it to be processed for caching.

When CacheQuickHandler is disabled, everything just drops into place, though some fine tuning might be required.

The current configuration for my use case of caching media for my Mastodon instance that's being retrieved from a horribly sluggish Minio backend looks like this:


        CacheQuickHandler off
        CacheRoot /var/cache/apache2/mod_cache_disk
        CacheMaxFileSize 10000000
        CacheDirLevels 2
        CacheDirLength 1
        CacheLock off
        CacheIgnoreCacheControl On
        CacheIgnoreQueryString On
        CacheStoreNoStore On
        CacheIgnoreHeaders Set-Cookie X-Amz-Request-Id

...and then:


        Require all granted
        ProxyPass http://:9000/
        ProxyPassReverse http://:9000/
        
               CacheEnable disk

25Gbit ethernet is complicated...

2020-02-10T22:47:00+00:00

We just spent about a week trying to put a bunch of systems into production that had been ordered with 25Gbit fiber interfaces. We had planned to collect those on two of our Arista 7050CX3, using 100GBit QSFP28 in 4 * 25GBit mode and MPO breakout cables to 4 * LC for the 25Gbit SFP28 end. So we cable everything up, configure our LACP channels on both ends, and ... nothing. All of the links stay down.

They do show a signal on the transciever though (at least on the switch side where we can look at optics information). An show interfaces et10/1-4 status says "notconnect" for all four subinterfaces. An show interfaces et10/1-4 phy displays an "errDisabled" on the phy layer. We are stumped.

Over the course of the next few days, we try several changes, to no avail. Directly connecting two Arista switches works though, as does a direct connection between two end hosts. We even swap everything down to 40G on the Arista side and 10G SFP+ in the end hosts, which turns out perfectly fine (so at least our cabling is correct).

At this point, support for the appliances we're trying to connect gives us credentials for shell access. It's a non-root user on what turns out as a normal Linux system, but at least I can see that it comes with QLogic Corp. FastLinQ QL45000 Series 25GbE controllers (for a short moment we had suspected we had the wrong controllers), and I can get some information by using ethtool. One of those is that ethtool reports the host interfaces as "25GBASE-KR", which tells me nothing. Someone on IRC mentions that "-KR" denotes an "electrical backplane" connection. Armed with those two small bits of information, I hit the search engines, and find this useful table in a document on the Marvell web site:

It's accompanied by the following text:

The –S short reach interfaces aim to support high-quality cables without
ForwardError Correction (FEC) to minimize latency. Full reach interfaces
aim to support the lowest possible cable or backplane cost and the longest
possible reach, which do require the use of FEC. FEC options include
BASE-R FEC (also referred to as Fire Code) and RS-FEC (also referred to
as Reed-Solomon).

There's two different, incompatible, error correction mechanisms on the bitstream layer of 25Gbit interfaces!? I didn't know that.

Since the default on Arista switches seems to be Reed-Solomon, and I don't have any way to configure a detail like that on the end host, we change the configuration on the Arista side:

interface et10/1-4
error-correction encoding fire-code

That's all. We do the same for three other interface groups, and all links work just excpected (except for one that apparently has a bad transciever in the end host). I call off the screen-sharing session with Arista support planned for five minutes later.

backing up lxc container snapshots, Amanda style

2019-11-10T15:29:00+00:00

I'm probably about the only person in the world using that kind of setup, but here we go:

I have an active Amanda (cache) installation that I use to back up various UNIX systems (to disk, with a weekly flush out to a tape rotation)
I run a system with lxc containers, using btrfs as storage backend

On btrfs, lxc containers are just subvolumes mounted into the host filesystem, and container snapshots are btrfs snapshots attached to the snapshots/ subdirectory of the container host volume.

So I'm running a simple script on the lxc host each night that cycles through all the containers and creates a snapshot named "amanda" for each of them - deleting the previous version if present. The main loop of the bash script looks more or less like this:

if [ -d /${lxdpool}/snapshots/${container}/amanda ]; then
 lxc delete ${container}/amanda
 sleep 2
fi
lxc snapshot ${container} amanda

Amanda can do incremental backups using GNU tar (in addition to a host of other options). One of the less obvious stumbling blocks with this is that GNU tar takes the device ID into account when calculating incrementals - and as each btrfs snapshot is a new device, the default configuration will back up all of the files in the snapshot every day, even if the file metadata is unchanged. So to make this setup work, Amanda needs a new dumptype with a tar configuration that ignores the device ID (tar option --no-check-device). The amanda.conf on my backup server now defines this in addition to the pre-existing defaults:

 define application-tool app_amgtar_snap { #
    comment "amgtar for btrfs snapshots"
    plugin "amgtar"
    property "ONE-FILE-SYSTEM" "yes"  #use '--one-file-system' option
    property "ATIME-PRESERVE" "yes"   #use '--atime-preserve=system' option
    property "CHECK-DEVICE" "no"      #use '--no-check-device' if set to "no"
    property "IGNORE" ": socket ignored$"  # remove some log clutter
    property append "IGNORE" "directory is on a different filesystem"
}

define dumptype dt_amgtar_snap { #
    comment "new dump type that uses the above application definition"
    program "APPLICATION"
    application "app_amgtar_snap"
}

 define dumptype comp-user-ssh-tar-lxd-snap { #
    global-ssh   # use global ssh transport configuration
    client_username "backup"
    program "GNUTAR"
    dt_amgtar_snap    # that's my new dumptype
    comment "partitions dumped with tar as lxd snapshot, using gnutar --no-device option"
    index
    priority low
    compress client fast
    exclude list "./rootfs/.amandaexclude"  # each container can have individual exclude lists in /.amandaexclude
}

All that's left now is to add entries to the Amanda disklist that are using my new dump type:

host.example.com        /lxdpool/snapshots/container1/amanda     comp-user-ssh-tar-lxd-snap
host.example.com        /lxdpool/snapshots/container2/amanda     comp-user-ssh-tar-lxd-snap

things that can go wrong when installing Smokeping

2019-11-08T23:46:00+00:00

Yeah, haven't done that in a long time:

(from https://mastodon.infra.de/@galaxis/103104946365972858 (cache))

adding a current FreeMiNT release into an existing EasyMiNT install on the Atari TT

2019-11-03T12:02:00+00:00

I spent this weekend installing EasyMiNT (cache) on my Atari TT, and then making it work with the Lightning VME USB board (cache).

Some more on the journey in getting there can be seen in these Fediverse threads:

Notes:

I haven't had much luck in installing EasyMiNT to anything other than the C: drive
The MiNT kernel provided with EasyMiNT is too old to be able to load the Lightning drivers, but since I had successfully installed the EasyMiNT distribution already, I wanted to upgrade it with a current FreeMiNT (cache) release.
Booting the current MiNT kernel (as of 1-19-73f) hangs after the "Installing BIOS keyboard" message. This thread (cache) on atari-forum.com recommends removing BIGDOS.PRG from the AUTO folder. Apparently BIGDOS is not required when using recent MiNT kernels anyway (I also got rid of WDIALOG.PRG while I was at it).

EasyMiNT installed on C: boots the kernel from C:\MINT\1-19-CUR. I didn't want to touch that working part of the setup, so I downloaded a full snapshot from https://bintray.com/freemint/freemint/snapshots/ that uses the snapshot version as MiNT SYSDIR (C:\MINT\1-19-73f for my build). Changing from the EasyMiNT kernel to the current MINT030.PRG in C:\AUTO\ then implicitly executes everything else from the corresponding SYSDIR.

As it turns out, the USB drivers included with the current FreeMiNT distribution are incompatible with those from the Lighting VME driver disk. The easiest way is to rename $SYSDIR\USB to something else and replace the directory with the files from the TT\MINT directory in the Lightning distribution - and then add a missing file (ETH.UDD) attached to this forum post (cache). Using the ETH.UDD provided with FreeMiNT does not work and leads to an "API Mismatch" message.

To keep using most of the EasyMiNT setup, I adapted the boot sequence and MINT.CNF (some hints on the general boot layout can be found in MiNTBootSequence) by replacing some of the sln links. The relevant sections of my current configuration looks like this (E: is my EasyMiNT ext2 filesystem):

# add some binaries provided by FreeMiNT, later referenced in PATH
sln c:/mint/1-19-73f/sys-root/bin              u:/sysbin
# GEM programs included in the FreeMiNT distribution
sln c:/mint/1-19-73f/sys-root/opt              u:/opt
sln c:/mint/1-19-73f/sys-root/share            u:/share
# EasyMINT links
sln e:/etc     u:/etc
sln e:/bin     u:/bin
sln e:/sbin    u:/sbin
sln e:/home    u:/home
sln e:/usr     u:/usr
sln e:/mnt     u:/mnt
sln e:/root    u:/root
sln e:/tmp     u:/tmp
# this line only works after removing the /usr/bin/xaaes symlink in EasyMiNT!
# with this, the EasyMiNT/SpareMiNT init script keeps starting XaAES without any further changes
sln c:/mint/1-19-73f/xaaes/xaloader.prg    u:/usr/bin/xaaes

# I've found that using TOS paths in MINT.CNF works better?
setenv PATH u:\sysbin,u:\bin,u:\usr\bin,u:\usr\sbin,u:\sbin,u:\c\mint\1-19-73f\xaaes

setenv TMPDIR u:\tmp

# provided by EasyMiNT, only works when the appropriate direcories on E: are linked in
exec u:\c\mint\bin\sh u:\c\mint\bin\fscheck.sh

setenv TZ 'Europe/Berlin'
exec u:\sbin\tzinit -l

# load Lightning USB drivers
exec u:\c\mint\1-19-73f\usb\loader.prg

# use SpareMiNT init system, as installed by EasyMiNT
INIT=u:\sbin\init

Linking in XALOADER.PRG via an sln link makes it easy to adapt the configuration to new releases. Most of the rest of the sln link tree comes from the MINT.CNF created by the EasyMiNT installer.

deleting stale VMware NSX controller instances

2019-07-06T17:07:00+00:00

When the installation of an VMware NSX controller fails and it's locked in the UI, you can just delete it from the API:

(from https://mastodon.infra.de/@galaxis/102376746230112269 (cache))

Apache 2.4 as a reverse proxy for Mastodon

2019-05-31T19:45:00+00:00

The standard setup for Mastodon is to use nginx as a reverse proxy. After one too many missing features I recently switched my installation over to using good old Apache.

There's an example Apache config (cache) in the unmaintained old documentation archive for Mastodon, and since I assume it's useless to try to update that, I'll quickly dump my current config here. There's no guarantee for correctness, but it currently seems to work for me. Note that this configuration does not do any caching for requests to static content retrieved through the reverse proxy.

The following Apache modules are used:

proxy
proxy_http
http2
proxy_http2
proxy_wstunnel
headers
socache_shmcb
ssl

General SSL configuration (personal preference, CipherSuite selection is probably going to age badly). TLS v1.3 is disabled since Ubuntu bionic ships an Apache version that's too old for that:



        SSLCertificateFile     
        SSLCertificateKeyFile  
        #   the referenced file can be the same as SSLCertificateFile
        #   when the CA certificates are directly appended to the server
        #   certificate for convinience.
        SSLCertificateChainFile 

        # SSLProtocol -all +TLSv1.2 +TLSv1.3
        SSLProtocol -all +TLSv1.2 +TLSv1.1
        SSLHonorCipherOrder on
        SSLCipherSuite ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:EECDH+AESGCM:AES256+EECDH:AES128+EECDH
        SSLCompression off
        SSLSessionTickets off
        SSLSessionCache "shmcb:logs/session-cache(512000)"
        SSLStaplingResponderTimeout 5
        SSLStaplingReturnResponderErrors off
        SSLUseStapling on
        SSLStaplingCache "shmcb:logs/stapling-cache(150000)"

        # needs to be generated first, see https://weakdh.org/sysadmin.html
        SSLOpenSSLConfCmd DHParameters /etc/ssl/dhparam.pem

Mastodon vhost configuration:


        ServerAdmin webmaster@example.com
        ServerName mastodon.example.com

        SSLEngine on

        Protocols h2 http/1.1

        # fetch static files directly from local file system (adapt to installation path)
        DocumentRoot /home/mastodon/live/public

        Header always set Strict-Transport-Security "max-age=31536000"

        
                Header always set Cache-Control "public, max-age=31536000, immutable"
                Require all granted
        

        
                Require all granted
        

        ProxyPreserveHost On
        RequestHeader set X-Forwarded-Proto "https"
        ProxyAddHeaders On

        # these files / pathes don't get proxied and are retrieved from DocumentRoot
        ProxyPass /500.html !
        ProxyPass /sw.js !
        ProxyPass /robots.txt !
        ProxyPass /manifest.json !
        ProxyPass /browserconfig.xml !
        ProxyPass /mask-icon.svg !
        ProxyPassMatch ^(/.*\.(png|ico)$) !
        ProxyPassMatch ^/(assets|avatars|emoji|headers|packs|sounds|system) !
        # everything else is either going to the streaming API or the web workers
        ProxyPass /api/v1/streaming ws://localhost:4000
        ProxyPassReverse /api/v1/streaming ws://localhost:4000
        ProxyPass / http://localhost:3000/
        ProxyPassReverse / http://localhost:3000/

        ErrorDocument 500 /500.html
        ErrorDocument 501 /500.html
        ErrorDocument 502 /500.html
        ErrorDocument 503 /500.html
        ErrorDocument 504 /500.html

The trailing / on the websocket ProxyPass directive is missing by design (it's there in the old example config): Some API requests seen in the wild will not match /api/v1/streaming/ and will get lost.

creating an iPXE boot floppy

2018-06-30T22:20:00+00:00

The iPXE open source boot firmware project provides an CD image that boots the iPXE binary using isolinux.

Over on the Fediverse, the topic of bootstraping a system from a floppy disk came up, and with the iPXE binary being a mere 330KB, there's really no reason why it shouldn't be possible to boot that from a floppy disk. And it actually does work, with a few simple steps (on a Debian-ish Linux):

format floppy disk and create FAT filesystem
```
fdformat /dev/fd0
mkfs -t fat /dev/fd0
```

get syslinux and install to floppy

apt install syslinux syslinux-utils
syslinux --install /dev/fd0

get iPXE ISO
```
curl -O http://boot.ipxe.org/ipxe.iso
```

mount both iPXE ISO and floppy, copy over required files, rename isolinux.cfg to syslinux.cfg

mkdir fd iso
mount /dev/fd0 fd
mount -o ro ipxe.iso iso
cp iso/ipxe.krn fd/
cp iso/boot.cat fd/
cp iso/isolinux.cfg fd/syslinux.cfg
umount fd
umount iso
rmdir fd iso

That's all! Take your floppy and boot a system

Once iPXE has been started, hit Ctrl-B to call the shell. If you have a DHCP server on your network and a web server with a bootable ISO image, it's just two iPXE commands:

dhcp
sanboot http:///.iso

SolidFire FDVA software repository downgrade

2017-12-21T15:11:00+00:00

We've been playing with a SolidFire flash storage cluster for some time, and recently wanted to update the nodes to the current ElementOS 10.1 release.

Unfortunately, our FDVA management node installation was borked, so we decided to just roll a new one from the current VM appliance template - easy.
As it turns out though, the FDVA appliance only ships with the latest software release files, and the individual SolidFire nodes check back for a repository with their current version before starting the update, which consequently fails (it's all very Ubuntu-ish):

admin@SF-7323:~$ sudo sfinstall 192.168.10.21 -u admin -p password -l
2017-12-20 17:27:52: sfinstall Release Version: 10.1.0.83 Revision:  Build date: 2017-11-23 01:27
2017-12-20 17:27:52: Checking connectivity to MVIP 192.168.10.21
2017-12-20 17:27:52: Successfully connected to cluster MVIP
2017-12-20 17:27:53: PrintRepositoryPackages failed: SolidFireApiError server=[192.168.10.10] method=[AptUpdate], params=[{'quiet': 2}] - error name=[xCheckFailure], 
message=[cmdResult={ rc=255 stdout="W: Failed to fetch http://192.168.10.10/fluorine-updates/ubuntu/dists/precise/main/binary-amd64/Packages  404  Not Found
[..]
W: Failed to fetch http://192.168.10.10/fluorine-updates/security-ubuntu/dists/precise-security/universe/binary-amd64/Packages  404  Not Found

The SolidFire docs don't really mention what to do from there, so we tinkered around for some time and found this:

Any older version of the repository can be fetched using the update-fdva tool with the currently used ElementOS release version as command line (version number can be seen on the cluster web UI or when asking the cluster nodes for their mnode repository using sfinstall). In our case, the active version was 9.2.0.43 -

admin@SF-7323:~$ sudo update-fdva 9.2.0.43
Get: 1 http://localhost precise Release.gpg [490 B]
Get: 2 http://localhost precise-updates Release.gpg [490 B]
[..]

This will fetch the 9.2.0.43 version of the SolidFire repository, but will also downgrade to the matching (old) versions of solidfire-fdva-tools and solidfire-python-framework...

admin@SF-7323:~$ dpkg -l | grep fdva
ii  solidfire-fdva-tools-fluorine-patch2-9.2.0.43               9.2.0.43                          SolidFire FDVA Tools 9 [fluorine-patch2]

...so we immediately reinstalled the current versions, using update-fdva again, this time with the current release version number:

admin@SF-7323:~$ sudo update-fdva 10.1.0.83

With all that in place, we could just run the update routine using the usual sfinstall command.

find obsolete packages on a Debian system

2017-07-08T08:40:00+00:00

After dist-upgrading a Debian system recently, I wondered which packages might have been left over from previous releases (the system in question has been through several dist-upgrades over its lifetime), even after running apt-get autoremove and deborphan. After dropping that question on Mastodon (cache), I got an answer pointing to apt-show-versions, which I didn't know of up to now.

This totally does what I've been looking for. From the man page:

NAME
       apt-show-versions - Lists available package versions with distribution

DESCRIPTION
       apt-show-versions parses the dpkg status file and the APT lists for the installed and available package
       versions and distribution and shows upgrade options within the specific distribution of the selected package.

       This is really useful if you have a mixed stable/testing environment and want to list all packages which are
       from testing and can be upgraded in testing.

Since I didn't have a package cache for apt-show-versions from the older release, all old packages are currently just shown with a No available version in archive comment. But since current packages are being tagged with the release, I can just exclude those with a simble grep:

# apt-show-versions | egrep -vc jessie
58

Intel D945GCLF2, ACPI Exception: AE_AML_INFINITE_LOOP

2017-05-10T21:41:00+00:00

Another of those "just so I find my own post the next time I'm looking for this" things...

After the replacing the CPU fan on an old Intel D945GCLF2 Atom board, I "optimized" the BIOS settings by enabling automatic fan control (instead of having the fan at a fixed speed).

Currently, I have lots of messages like this in my kernel log, and a kworker thread using 100% CPU:

Copy to clipboard

ACPI Error: Method parse/execution failed [\_SB_.PCI0.LPC_.SMBR] (Node ffff88007ec3f900), AE_AML_INFINITE_LOOP (20140424/psparse-536)
ACPI Error: Method parse/execution failed [\_SB_.PCI0.LPC_.INIT] (Node ffff88007ec3f928), AE_AML_INFINITE_LOOP (20140424/psparse-536)
ACPI Error: Method parse/execution failed [\_GPE._L00] (Node ffff88007ec35bd0), AE_AML_INFINITE_LOOP (20140424/psparse-536)
ACPI Exception: AE_AML_INFINITE_LOOP, while evaluating GPE method [_L00] (20140424/evgpe-580)

So, an hour or so of searching later, I finally hit this comment on the Novell bugzilla (cache) - and then I promptly remembered that I used to have known about this problem, and that it was exactly the reason why the fan was set to a fixed speed:

In BIOS I have DISABLED auto fan speed. It is now set at 90%. It seems to have fixed it

sendmail MIME conversion vs. DMARC+DKIM

2017-04-15T13:38:00+00:00

I've recently tried to reconfigure a mailinglist that I run on one of my systems to make less problems with recipients that use DMARC.

To that end, I wanted to implement the first option mentioned in the corresponding DMARC FAQ for mailinglist administrators (cache): Don't do any changes to the message body and headers potentially covered by a DKIM signature.

On the mailinglist configuration side, this means not adding a list tag to the message subject, and not changing the body with an additional header or footer. (Adding the usual RFC2369/RFC2919 list headers is no problem.)

After doing that it turned out that my setup, using sendmail, still changed the body on some messages, thanks to the automatic MIME autoconversion that sendmail does.

Getting rid of that actually required some digging into sendmail configuration and documentation:

Responsible for message delivery to local programs is the aptly named "prog" mailer. Unfortunately, as far as the m4 configuration statements are concerned, all the variables for this mailer are called "SHELL"... The default flags for the prog/shell mailer are hardcoded to "[eu9]", according to the README (though these defaults are changed in some of the OSTYPE definitions in the m4 macro collection):

sendmail cf.README wrote:

LOCAL_SHELL_FLAGS [eu9] The flags used by the shell mailer. The flags lsDFM are always included.

The meaning of the individual flags is documented in the OP manual:

sendmail OP manual wrote:

7
   Strip all output to seven bits. This is the default if the L flag is set. Note that clearing this option is not sufficient to get full eight bit data passed through sendmail. If the 7 option is set, this is essentially always set, since the eighth bit was stripped on input. Note that this option will only impact messages that didn't have 8->7 bit MIME conversions performed.
8
   If set, it is acceptable to send eight bit data to this mailer; the usual attempt to do 8->7 bit MIME conversions will be bypassed.
9
   If set, do limited 7->8 bit MIME conversions. These conversions are limited to text/plain data.

So it seems I want to get rid of the 9 in the LOCAL_SHELL_FLAGS, and replace it by a 8...

Adding the following two statements to my sendmail m4 configuration source does exactly that:

sendmail.mc

Copy to clipboard

dnl # disable MIME-Autoconversion for prog mailer
MODIFY_MAILER_FLAGS(`SHELL', `-9')
MODIFY_MAILER_FLAGS(`SHELL', `+8')

raspbian jessie - rsyslogd-2007: action 'action 17' suspended, next retry ...

2017-04-10T19:45:00+00:00

On a headless Raspberry Pi running raspbian/jessie, the /var/log/messages file is filling up with entries like these:

 rsyslogd-2007: action 'action 17' suspended, next retry is [..date..] [ try http://www.rsyslog.com/e/2007 ]

It seems this message is generated when rsyslogd isn't able to deliver syslog messages to one of the destinations in rsyslog.conf

In the case a raspbian, it's obviously the entry at the end of the config that tries to pipe messages to |/dev/xconsole - which doesn't exist on a system that doesn't run X11...

The messages disappear after commenting out or deleting the corresponding lines:

/etc/rsyslog.conf

Copy to clipboard

#daemon.*;mail.*;\
#       news.err;\
#       *.=debug;*.=info;\
#       *.=notice;*.=warn       |/dev/xconsole

I really should file a bug report for this...

Splunk eval vs. variable names with dashes

2017-04-05T20:54:00+00:00

I'm pretty certain I used to know this - but for the next time I'm putting this into a search engine and don't find it in the Splunk docs:

One of our data sources writes structured data into our Splunk installation which contains variable names with dashes - in this particular case, access-time

It's no problem using such a variable in a lot of Splunk operations, but it fails in an eval, as it will be interpreted as a mathematical operation (access minus time).

There's two options to work around that:

the one mentioned in the Splunk documentation: Put the variable name in single quotes, i.e. | eval newtime='access-time' - constant
the other one is to simply rename the variable before working on it: | rename access-time AS accesstime | eval newtime=accesstime - constant

downgrading Android apps using data from TWRP backups

2017-03-27T22:41:00+00:00

Mostly as a reminder to myself when I'm looking to solve this kind of problem the next time: Since the March 22, 2017 version of the FortiClient VPN Android app kept crashing on my mobile (still running the last Cyanogenmod 13 snapshot) as soon as I tried to switch away to the launcher, I wanted to downgrade the app.

Unfortunately, there's no copy on apkmirror.com or F-Droid, and I don't know about any other reasonably trustworthy sources. I also already had removed and reinstalled the app, so recovering the old version on the phone didn't seem an option either.

Fortunately, I take TWRP backups now and then, so I tried looking at one of those. For once, having unencrypted backups turned out real convenient: A TWRP data.ext4.win file is just a tar.gz, so I was able recover the app/com.fortinet.forticlient_vpn-1/base.apk file (using 7Zip on Windows), and copy that over to my phone. After uninstalling the current version of the FortiClient app, I just reinstalled the program with the CM file manager using the restored base.apk as a source. Done.

Cisco ASA logging: Disable hiding of usernames in failed admin logins

2017-03-23T15:28:00+00:00

Cisco ASA firewalls don't log, by default, the username used in a failed administrator login. Instead, the login is masked out using "*" characters:

%ASA-6-113005: AAA user authentication Rejected : reason = AAA failure : server = 10.1.1.1 : user = ***** : user IP = 192.168.0.10

The rationale is that users sometimes enter their password instead of the username, and the password will then end up in logs. As we're using two-factor authentication for admin logins, that doesn't apply to us.

That behaviour was actually tracked as a bug in Cisco's bug database (cache), and while the article mentions that a command was introduced to change this behaviour, the command itself isn't mentioned.

After some fiddling on the ASA command line I found this statement:

no logging hide username

The corresponding button in the ASDM GUI is in Device Management -> Logging -> Syslog Setup: "Hide username if its validity cannot be determined"

so I didn't notice that my OpenBSD vserver had broken IPv6 for quite some time...

2017-02-19T13:37:00+00:00

...until I had a look at the DNS server log, which showed errors contacting other servers via IPv6.

The hoster I'm using has a somewhat strange IPv6 setup where you get a /64 for your system, but the default gateway is just fe80::1 - when I originally set up the system, I put that into /etc/mygate whithout thinking much about it.

This initially was ok for quite some time, but it seems the default route vanished at some point. (In retrospect I don't quite understand why the setup ever worked at all, as the lo0 lookback interface has fe80::1 auto-assigned too...)

Then I remembered that fe80:: carries interface tags, since it exists on any IPv6-enabled interface, and the OS needs some way to decide which fe80:: it has to deal with right now.

Edited /etc/mygate accordingly, and things are back to normal (vio is OpenBSD's VirtIO network device driver, so my virtual ethernet device is vio0):

fe80::1%vio0